U.S. patent application number 12/394765 was filed with the patent office on 2009-06-25 for image coding/decoding method, image coding apparatus and image decoding apparatus.
This patent application is currently assigned to NTT DoCoMo, Inc. Invention is credited to Minoru Etoh, Shunichi Sekiguchi.
Application Number | 20090161972 12/394765 |
Document ID | / |
Family ID | 26605459 |
Filed Date | 2009-06-25 |
United States Patent
Application |
20090161972 |
Kind Code |
A1 |
Sekiguchi; Shunichi ; et
al. |
June 25, 2009 |
IMAGE CODING/DECODING METHOD, IMAGE CODING APPARATUS AND IMAGE
DECODING APPARATUS
Abstract
An image coding method is provided, in which the method includes
the steps of: extracting edge information which represents an edge
part of an original image; obtaining density information of an edge
smoothed image from the original image by smoothing the edge part;
obtaining coded edge information by coding the edge information
according to a first coding algorithm; obtaining coded density
information by coding the density information of the edge smoothed
image according to a second coding algorithm; and sending the coded
edge information and the coded density information as coded
information to an image decoding apparatus.
Inventors: |
Sekiguchi; Shunichi;
(Yamato-shi, JP) ; Etoh; Minoru; (Yokohama-shi,
JP) |
Correspondence
Address: |
OBLON, SPIVAK, MCCLELLAND MAIER & NEUSTADT, P.C.
1940 DUKE STREET
ALEXANDRIA
VA
22314
US
|
Assignee: |
NTT DoCoMo, Inc
Chiyoda-ku
JP
|
Family ID: |
26605459 |
Appl. No.: |
12/394765 |
Filed: |
February 27, 2009 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10003113 |
Dec 6, 2001 |
7532762 |
|
|
12394765 |
|
|
|
|
Current U.S.
Class: |
382/233 ;
382/199 |
Current CPC
Class: |
G06T 2207/10016
20130101; H04N 19/61 20141101; H04N 19/80 20141101; H04N 19/44
20141101; G06T 7/12 20170101; G06T 9/20 20130101; G06T 2207/20052
20130101 |
Class at
Publication: |
382/233 ;
382/199 |
International
Class: |
G06K 9/36 20060101
G06K009/36 |
Foreign Application Data
Date |
Code |
Application Number |
Dec 7, 2000 |
JP |
2000-373462 |
Mar 14, 2001 |
JP |
2001-072904 |
Claims
1. An image coding/decoding method in which an image coding
apparatus sends coded information which is obtained by coding an
original image to an image decoding apparatus and said image
decoding apparatus decodes said coded information to obtain a
reproduced image, said method comprising the steps of: extracting
edge information which is binary information representing an edge
part of said original image; obtaining density information of an
edge unsharpened image from said original image by unsharpening
said edge part using said edge information; obtaining coded edge
information by coding said edge information according to a first
coding algorithm; obtaining coded density information by coding
said density information of said edge unsharpened image according
to a second coding algorithm; sending said coded edge information
and said coded density information as said coded information to
said image decoding apparatus; and said image decoding apparatus
performing the steps of: obtaining said edge information by
decoding said coded edge information according to a first decoding
algorithm corresponding to said first coding algorithm; obtaining
said density information of said edge unsharpened image by decoding
said coded density information according to a second decoding
algorithm corresponding to said second coding algorithm; obtaining
said reproduced image from said density information of said edge
unsharpened image by restoring said edge part of said edge
unsharpened image by using said edge information; wherein said
second coding algorithm and said second decoding algorithm are
based on a standard coding method using a discrete cosine
transform.
2. The image coding/decoding method as claimed in claim 1, said
image coding apparatus unsharpening said edge part by performing,
while scanning said original image pixel by pixel, the steps of:
performing first matrix operation by using a first block density
information vector and a unsharpening matrix, wherein said first
block density information vector is obtained by arranging density
information of each pixel included in a first block, said first
block includes a pixel in said edge part or in a near region of
said edge part and includes pixels in a surrounding region around
said pixel, and order of said first block density information
vector corresponds to the number of pixels in said first block, and
wherein said unsharpening matrix includes coefficients used for
edge unsharpening which operate on density information of each
pixel in said first block; obtaining unsharpened density
information of each pixel by overlaying density information of each
pixel in said first block obtained by performing said first matrix
operation on each pixel while scanning said original image pixel by
pixel.
3. The image coding/decoding method as claimed in claim 2, said
image decoding apparatus restoring said edge part of said edge
unsharpened image by performing, while scanning said edge
unsharpened image pixel by pixel, the steps of: performing second
matrix operation by using a second block density information vector
and a restoring matrix which is an inverse matrix of said
unsharpening matrix, wherein said second block density information
vector is obtained by arranging density information of each pixel
included in a second block, said second block includes a pixel in
said edge part or in a near region of said edge part and pixels in
said surrounding region, and order of said second block density
information vector corresponds to the number of pixels in said
second block; and obtaining restored density information of each
pixel by overlaying density information of each pixel in said
second block obtained by performing said second matrix operation on
each pixel while scanning said edge unsharpened image pixel by
pixel.
4. The image coding/decoding method as claimed in claim 1, said
image coding apparatus unsharpening said edge part by performing
the steps of: obtaining density information x' of a pixel of said
edge part of said edge unsharpened image according to a first
equation x'=(1-.lamda.)x+.times..lamda.C, wherein .lamda. is a
positive constant, x is density information of said pixel of said
original image, and C is surrounding density information
representing density state of a surrounding region of said
pixel.
5. An image coding apparatus comprising: an edge extracting part
for extracting edge information which is binary information
representing an edge part of an original image; an edge
unsharpening part for obtaining density information of an edge
unsharpened image from said original image by unsharpening said
edge part using said edge information; a first coding part for
obtaining coded edge information by coding said edge information
according to a first coding algorithm; a second coding part for
obtaining coded density information by coding said density
information of said edge unsharpened image according to a second
coding algorithm; wherein said coded edge information and said
coded density information are coded information of said original
image, and wherein said second coding algorithm is based on a
standard coding method using a discrete cosine transform.
6. The image coding apparatus as claimed in claim 5, said edge
unsharpening part including a density information correction part
for correcting density information of each pixel such that
variation of density levels represented by density information of
pixels which are arranged across said edge part in a near region of
said edge part of said original image is lowered.
7. The image coding apparatus as claimed in claim 5, said edge
unsharpening part comprising: a unsharpening matrix generation part
for generating, for each block which includes said edge part or a
near region of said edge part, a unsharpening matrix which is used
for matrix operation with a block density information vector,
wherein said block density information vector is obtained by
arranging density information of each pixel included in a block,
and order of said block density information vector corresponds to
the number of pixels in said block, and wherein said unsharpening
matrix includes coefficients used for edge unsharpening which
operate on density information of each pixel in said edge part or
in said near region in said block; and a matrix operation part for
obtaining unsharpened density information of each pixel in said
block by performing matrix operation by using said unsharpening
matrix and said block density information vector.
8. The image coding apparatus as claimed in claim 5, said edge
unsharpening part comprising: a pixel judgement part for judging
whether a pixel exists in said edge part or in a near region of
said edge part while scanning said original image pixel by pixel; a
matrix operation part for performing, when said pixel exists in
said edge part or in said near region, matrix operation by using a
block density information vector and a unsharpening matrix, wherein
said block density information vector is obtained by arranging
density information of each pixel included in a block, said block
includes said pixel and pixels in a surrounding region around said
pixel, and order of said block density information vector
corresponds to the number of pixels in said block, and wherein said
unsharpening matrix includes coefficients used for edge
unsharpening which operate on density information of each pixel in
said block; an operation part for obtaining unsharpened density
information of each pixel by overlaying density information of each
pixel in said block obtained by performing said matrix operation on
each pixel while scanning said original image pixel by pixel.
9. The image coding apparatus as claimed in claim 5, wherein said
edge unsharpening part obtains density information x' of a pixel of
said edge part of said edge unsharpened image according to an
equation x'=(1-.lamda.)x+.lamda.C, wherein .lamda. is a positive
constant, x is density information of said pixel of said edge part
of said original image, and C is surrounding density information
representing density state of a surrounding region of said
pixel.
10. An image decoding apparatus which decodes coded information
which includes coded edge information obtained by coding edge
information representing an edge part of an original image and
coded density information representing an edge unsharpened image
obtained by unsharpening said edge part of said original image
using said edge information, said image decoding apparatus
comprising: a first decoding part for obtaining said edge
information which is binary information representing said edge part
by decoding said coded edge information according to a first
decoding algorithm; a second decoding part for obtaining density
information of said edge unsharpened image by decoding said coded
density information according to a second decoding algorithm; an
edge restoring part for restoring said edge part of said edge
unsharpened image by using said edge information such that a
reproduced image is obtained, wherein said second decoding
algorithm is based on a standard coding method using a discrete
cosine transform.
11. The image decoding apparatus as claimed in claim 10, said edge
restoring part including a density information correction part for
correcting density information of each pixel of said edge
unsharpened image such that variation of density levels represented
by density information of pixels which are arranged across said
edge part in a near region of said edge part of said edge
unsharpened image is increased.
12. The image decoding apparatus as claimed in claim 10, said edge
restoring part comprising: a restoring matrix generation part for
generating, for each block in said edge part or a near region of
said edge part in said edge unsharpened image, a restoring matrix
which is used for matrix operation with a block density information
vector, wherein said block density information vector is obtained
by arranging density information of each pixel included in a block,
and order of said block density information vector corresponds to
the number of pixels in said block, and wherein said restoring
matrix includes coefficients used for edge restoring which operate
on density information of each pixel in said edge part or in a near
region of said edge part in said block; and a matrix operation part
for obtaining restored density information of each pixel in said
block by performing said matrix operation by using said restoring
matrix and said block density information vector.
13. The image decoding apparatus as claimed in claim 12, wherein
said restoring matrix generation part generates an inverse matrix
of a unsharpening matrix as said restoring matrix in which said
unsharpening matrix is used for obtaining density information of
said edge unsharpened image which is decoded from said coded
density information.
14. The image decoding apparatus as claimed in claim 10, said edge
restoring part comprising: a pixel judgement part for judging
whether a pixel exists in said edge part represented by said edge
information or in a near region of said edge part while scanning
said edge unsharpened image pixel by pixel; a matrix operation part
for performing, when said pixel exists in said edge part or in said
near region, matrix operation by using a block density information
vector and a restoring matrix, wherein said block density
information vector is obtained by arranging density information of
each pixel included in a block, said block includes said pixel and
pixels in a surrounding region around said pixel, and order of said
block density information vector corresponds to the number of
pixels in said block, and wherein said restoring matrix includes
coefficients used for edge restoring which operate on density
information of each pixel in said block; an operation part for
obtaining restored density information of each pixel by overlaying
density information of each pixel in said block obtained by
performing said matrix operation on each pixel while scanning said
edge unsharpened image pixel by pixel.
15. The image decoding apparatus as claimed in claim 14, wherein
said restoring matrix is an inverse matrix of a unsharpening matrix
in which said unsharpening matrix is used for obtaining density
information of said edge unsharpened image which is decoded from
said coded density information.
Description
CROSS REFERENCE
[0001] This application is a continuation of and is based upon and
claims the benefit of priority under 35 U.S.C. .sctn. 120 for U.S.
Ser. No. 10/003,113, filed Dec. 6, 2001, the entire contents of
each which are incorporated herein by reference. U.S. Ser. No.
10/003,113 claims the benefit of priority under 35 U.S.C. .sctn.
119 from Japanese Patent Application Nos. 2001-072904, filed Mar.
14, 2001 and 2000-373462, filed Dec. 7, 2000.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The present invention relates to an image coding/decoding
method used for coding original images such as a moving image and a
still image, and decoding coded images. More particularly, the
present invention relates to an image coding/decoding method which
can decrease noise for decoding coded images while applying a
standardized algorithm such as MPEG-1/2/4 and the like.
[0004] In addition, the present invention relates to an image
coding apparatus and an image decoding apparatus which perform
coding and decoding of images according to the image
coding/decoding method.
[0005] 2. Description of the Related Art
[0006] Conventionally, in standard image/moving image coding
methods such as JPEG and MPEG series, discrete cosine transform
(DCT) has been used for various reasons. Especially, these coding
methods has been adopted from the viewpoint of adaptability to
image properties.
[0007] Normally, when representing an image in the spatial
frequency region, there is a property that power concentrates on a
low frequency region. The DCT is a method in which a block formed
by 8.times.8 pixels in an image signal (density information) space
is orthogonally transformed so that the block is broken down to a
combination of predetermined base sets and coefficients of
constituent base are obtained. The characteristic of DCT is that
degree of bias of coefficients, that is, degree of bias of
frequency components can be increased. Especially, compression
efficiency can be increased by performing adaptive bit distribution
since the degree of concentration is higher in the low frequency
region and the low frequency region is visually important.
[0008] On the other hand, when coding an image at a very low bit
rate, quantization becomes rough so that reproducibility of
coefficients becomes worse. Thus, a problem occurs in that the base
which is important for representing image signal can not be
reconstructed. In addition, DCT performs processing on the region
closed in the 8.times.8 pixel blocks, distortion of coding tends to
appear remarkably on boundaries of the block. Due to this tendency,
block distortion occurs so that components which are not visually
included in the signal appears on the image. Thus, it is detected
as noise which is very conspicuous. Especially, in waveform parts
where luminance (density) changes abruptly such as in step edges,
many bases become necessary for reproducing the image faithfully.
However, code assignment for coefficients corresponding to high
frequency component is thought to be less important than code
assignment for coefficients corresponding to low frequency
component in consideration of visual weight. Thus, high frequency
coefficients important for reproducing the edge are lost. Due to
such processing, conspicuous noise which is intrinsic to DCT
appears in the edge part so that image degradation occurs.
[0009] In the light of the above-mentioned problem, an image coding
apparatus has been proposed in which DCT and differential
pulse-code modulation (DPCM) are switched adaptively according to
image patterns (for example, Japanese laid-open patent application
No. 5-276506). This conventional image coding apparatus avoids the
signal representation problem of DCT by using DPCM for processing
the edge part instead of using DCT.
[0010] However, according to the conventional image coding
apparatus, the standard coding method using the existing DCT as-is
can not be applied. Therefore, some kind of applicable coding
method other than the standard coding method is necessary. However,
since there are many products which comply with the standard coding
method using DCT, it is important to be able to perform coding and
decoding of an image by using the standard coding method.
SUMMARY OF THE INVENTION
[0011] A first object of the present invention is to provide an
image coding/decoding method in which an decoded image having small
distortion can be obtained even when the standard coding/decoding
method using DCT is applied.
[0012] A second object of the present invention is to provide an
image coding apparatus and an image decoding apparatus which
perform coding and decoding of an image according to the image
coding/decoding method.
[0013] The above object is achieved by an image coding/decoding
method in which an image coding apparatus sends coded information
which is obtained by coding an original image to an image decoding
apparatus and the image decoding apparatus decodes the coded
information to obtain a reproduced image, the image coding
apparatus performing the steps of:
[0014] extracting edge information which represents an edge part of
the original image;
[0015] obtaining density information of an edge smoothed image from
the original image by smoothing the edge part;
[0016] obtaining coded edge information by coding the edge
information according to a first coding algorithm;
[0017] obtaining coded density information by coding the density
information of the edge smoothed image according to a second coding
algorithm;
[0018] sending the coded edge information and the coded density
information as the coded information to the image decoding
apparatus; and
[0019] the image decoding apparatus performing the steps of:
[0020] obtaining the edge information by decoding the coded edge
information according to a first decoding algorithm corresponding
to the first coding algorithm;
[0021] obtaining the density information of the edge smoothed image
by decoding the coded density information according to a second
decoding algorithm corresponding to the second coding
algorithm;
[0022] obtaining the reproduced image from the density information
of the edge smoothed image by sharpening the edge part of the edge
smoothed image by using the edge information.
[0023] According to the image coding/decoding method, the image
coding apparatus extracts the edge part of the original image and
obtains the edge smoothed image by smoothing the edge part of the
original image. Then, the image coding apparatus sends coded
information of the original image to the image decoding apparatus
in which the coded information includes coded edge information
which is obtained from edge information representing the edge part
according to a first coding algorithm and coded density information
which is obtained from density information of the edge smoothed
image according to a second coding algorithm.
[0024] The image decoding apparatus sharpens the edge part of the
edge smoothed image obtained from the coded density information
according to a second decoding algorithm by using the edge
information obtained from the coded edge information according to a
first decoding algorithm. Then, a reproduced image is obtained from
the edge smoothed image by sharpening the edge part.
[0025] Since the edge part is smoothed in the edge smoothed image,
relatively high spatial frequency components decreases. Thus, even
when the standard coding method using DCT is used as the second
coding algorithm, information which is lost by coding
decreases.
[0026] The first coding algorithm used for coding the edge
information and the second coding algorithm used for coding density
information of the edge smoothed image may be or not may be the
same.
[0027] The edge information may be any information as long as the
position (pixel) of the edge part can be identified.
[0028] The density information is information which represents
state of image density such as density or luminance (or brightness)
and the form of the density information is not limited to a
specific form.
[0029] From the viewpoint of being able to perform the smoothing
process of the edge part of the original image relatively easily,
the image coding apparatus may smooth the edge part by performing,
while scanning the original image pixel by pixel, the steps of:
[0030] performing first matrix operation by using a first block
density information vector and a smoothing matrix, wherein the
first block density information vector is obtained by arranging
density information of each pixel included in a first block, the
first block includes a pixel in the edge part or in a near region
of the edge part and includes pixels in a surrounding region around
the pixel, and order of the first block density information vector
corresponds to the number of pixels in the first block, and wherein
the smoothing matrix includes coefficients used for edge smoothing
which operate on density information of each pixel in the first
block;
[0031] obtaining smoothed density information of each pixel by
overlaying density information of each pixel in the first block
obtained by performing the first matrix operation on each pixel
while scanning the original image pixel by pixel.
[0032] According to the image coding/decoding method, the density
information of the edge smoothed image can be obtained by matrix
operation using the smoothing matrix (which is a square matrix) and
the block density information vector.
[0033] From the viewpoint of being able to sharpen the edge part of
the edge smoothed image easily, the image decoding apparatus may
sharpen the edge part of the edge smoothed image by performing,
while scanning the edge smoothed image pixel by pixel, the steps
of:
[0034] performing second matrix operation by using a second block
density information vector and a sharpening matrix which is an
inverse matrix of the smoothing matrix, wherein the second block
density information vector is obtained by arranging density
information of each pixel included in a second block, the second
block includes a pixel in the edge part or in a near region of the
edge part and pixels in the surrounding region, and order of the
second block density information vector corresponds to the number
of pixels in the second block; and [0035] obtaining sharpened
density information of each pixel by overlaying density information
of each pixel in the second block obtained by performing the second
matrix operation on each pixel while scanning the edge smoothed
image pixel by pixel.
[0036] According to the image coding/decoding method, the density
information of each pixel which is sharpened can be obtained by
matrix operation using the block density information and the
sharpening matrix which is an inverse matrix of the smoothing
matrix.
[0037] From the viewpoint of being able to obtain more accurate
reproduced image in sharpening process, the image coding apparatus
may smooth the edge part by performing the steps of:
[0038] obtaining density information x' of a pixel of the edge part
of the edge smoothed image according to a first equation
x'=(1-.lamda.)x+.lamda.C, wherein .lamda. is a positive constant, x
is density information of the pixel of the original image, and C is
surrounding density information representing density state of a
surrounding region of the pixel.
[0039] The surrounding density information C is determined on the
basis of density information of surrounding pixels of the pixel.
The surrounding density information C is not limited to specific
information as long as it represents state of density around the
pixel. For example, a mean value of density information of
surrounding pixels, total added value, added value of weighted
density information or the mean value of the added value of
weighted density information or the like can be used as C..lamda.
in the above equation depends on how to determine the surrounding
density information C.
[0040] From the viewpoint of reproducing image from the smoothed
image smoothed by x'=(1-.lamda.)x+.lamda.C, the image decoding
apparatus may sharpen the edge part of the edge smoothed image by
using a predetermined equation according to a steepest-descent
method, the predetermined equation being defined on the basis of
the relationship between the density information x' and sharpened
density information formulated by the first equation.
[0041] The predetermined equation may be
e ( X ) = ( X + 1 1 - .lamda. ( .lamda. C ( n ) - x ' ) ) 2
##EQU00001##
[0042] in which C(n) is the surrounding density information for a
pixel having density information x' and n is a repetition count
number, and a value of X which minimizes e(X) is obtained by the
steepest-descent method and the value of X becomes density
information of a pixel after sharpened.
[0043] From the viewpoint of providing a concrete method of the
steepest-descent method, in a process according to the
steepest-descent method, X is obtained as a convergence value of a
recurrence formula
X ( n + 1 ) = X ( n ) - G * , .differential. e .differential. X '
##EQU00002##
[0044] wherein G is a constant.
[0045] The second object can be achieved by an image coding
apparatus including:
[0046] an edge extracting part for extracting edge information
which represents an edge part of an original image;
[0047] an edge smoothing part for obtaining density information of
an edge smoothed image from the original image by smoothing the
edge part;
[0048] a first coding part for obtaining coded edge information by
coding the edge information according to a first coding
algorithm;
[0049] a second coding part for obtaining coded density information
by coding the density information of the edge smoothed image
according to a second coding algorithm;
[0050] wherein the coded edge information and the coded density
information are coded information of the original image.
[0051] Accordingly, decoding of the original image becomes possible
by using the coded edge information and the coded density
information obtained as the coded information.
[0052] When decoding only coded density information, the edge
smoothed image is reproduced.
[0053] From the viewpoint of providing a concrete method for
obtaining the edge smoothed image, the edge smoothing part may
include a density information correction part for correcting
density information of each pixel such that variation of density
levels represented by density information of pixels which are
arranged across the edge part in a near region of the edge part of
the original image is lowered.
[0054] Since the variation of density levels represented by the
density information of pixels arranged across the edge part in the
near region of the edge part of the original image becomes small,
abrupt change of the density levels can be eliminated so that the
edge part is smoothed.
[0055] From the viewpoint of providing a further concrete method
for obtaining the edge smoothed image, the density information
correction part may includes:
[0056] a mean value calculation part for calculating a mean value
of the density levels in a predetermined region; and
[0057] a density level judgement part for judging whether the
density level of a pixel is higher or lower than the mean value for
each pixel in the near region;
[0058] wherein density information is corrected for a pixel in
which the density level is higher than the mean value such that the
density level is lowered, and density information is corrected for
a pixel in which the density level is lower than the mean value
such that the density level is increased.
[0059] The size and the shape of the predetermined region are not
limited as long as the predetermined region includes the near
region of the edge part. When applying a standardized coding
algorithm using DCT as the second coding algorithm, it is
preferable to use a DCT block (8.times.8 pixel) which is a process
unit of DCT as the predetermined region.
[0060] From the viewpoint of providing no change in the whole
density state of the original image, the density information
correction part corrects density information of each pixel in the
near region such that the mean value of the density levels does not
change.
[0061] According to such process, even when correction of density
information is performed on each pixel for each block, the mean
value of the density levels represented by the density information
of pixels in a block is not changed.
[0062] From the viewpoint of being able to perform edge smoothing
process of the original image relatively easily, the edge smoothing
part may includes:
[0063] a smoothing matrix generation part for generating, for each
block which includes the edge part or a near region of the edge
part, a smoothing matrix which is used for matrix operation with a
block density information vector, wherein the block density
information vector is obtained by arranging density information of
each pixel included in a block, and order of the block density
information vector corresponds to the number of pixels in the
block, and wherein the smoothing matrix includes coefficients used
for edge smoothing which operate on density information of each
pixel in the edge part or in the near region in the block; and
[0064] a matrix operation part for obtaining smoothed density
information of each pixel in the block by performing matrix
operation by using the smoothing matrix and the block density
information vector.
[0065] According to the image coding apparatus, smoothed density
information of all pixels in the block can be obtained by
performing matrix operation using the block density information
vector and the smoothing matrix for each predetermined block.
[0066] As the predetermined block, a block including any number of
pixels can be used. When a standardized coding algorithm using DCT
is applied as the second coding algorithm, it is preferable to use
a DCT block (8.times.8 pixel) which is a process unit of DCT as the
predetermined block.
[0067] Since coefficients used for edge smoothing are arranged in
the smoothing matrix generated for each block such that the
coefficients operate on density information of each pixel in the
edge part or in the near region in the block, the arrangement
depends on the position of the edge part or the near region
included in the block.
[0068] When performing the smoothing process for each predetermined
block, there is a possibility that density information on a
boundary of the blocks may be discontinued. From the viewpoint of
preventing this problem, the edge smoothing part may include:
[0069] a pixel judgement part for judging whether a pixel exists in
the edge part or in a near region of the edge part while scanning
the original image pixel by pixel;
[0070] a matrix operation part for performing, when the pixel
exists in the edge part or in the near region, matrix operation by
using a block density information vector and a smoothing matrix,
wherein the block density information vector is obtained by
arranging density information of each pixel included in a block,
the block includes the pixel and pixels in a surrounding region
around the pixel, and order of the block density information vector
corresponds to the number of pixels in the block, and wherein the
smoothing matrix includes coefficients used for edge smoothing
which operate on density information of each pixel in the
block;
[0071] an operation part for obtaining smoothed density information
of each pixel by overlaying density information of each pixel in
the block obtained by performing the matrix operation on each pixel
while scanning the original image pixel by pixel.
[0072] According to the image coding apparatus, since matrix
operation and pixel overlaying are performed while scanning the
original image pixel by pixel, the process can be performed
continuously on the original image pixel by pixel, wherein the
block density information and the smoothing matrix are used in the
matrix operation.
[0073] From the viewpoint of providing a concrete configuration of
the pixel judgement part, the pixel judgement part may include:
[0074] a distance conversion part for generating distance
information representing distances between the edge part and each
pixel; and
[0075] a distance judgment part for judging whether the distance
information for each pixel is equal to or smaller than a
predetermined value;
[0076] wherein, when the distance information is judged to be equal
to or smaller than the predetermined value, it is judged that a
pixel corresponding to the distance information exists in the edge
part or in the near region.
[0077] The second object can also be achieved by an image decoding
apparatus which decodes coded information which includes coded edge
information representing an edge part of an original image and
coded density information representing an edge smoothed image, the
image decoding apparatus including:
[0078] a first decoding part for obtaining edge information
representing the edge part by decoding the coded edge information
according to a first decoding algorithm;
[0079] a second decoding part for obtaining density information of
the edge smoothed image by decoding the coded density information
according to a second decoding algorithm;
[0080] an edge sharpening part for sharpening the edge part of the
edge smoothed image by using the edge information such that a
reproduced image is obtained.
[0081] According to the image decoding apparatus, by determining
the first decoding algorithm and the second decoding algorithm such
that they correspond to the first coding algorithm and the second
coding algorithm respectively, the coded information obtained by
the image coding apparatus can be properly decoded.
[0082] From the viewpoint of providing a concrete method of
sharpening the edge smoothed image, the edge sharpening part may
include a density information correction part for correcting
density information of each pixel of the edge smoothed image such
that variation of density levels represented by density information
of pixels which are arranged across the edge part in a near region
of the edge part of the edge smoothed image is increased.
[0083] Since the variation of density levels represented by the
density information of pixels arranged across the edge part in the
near region of the edge part of the edge smoothed image becomes
large, abrupt change of the density levels appears so that the edge
part is sharpened.
[0084] According to the image decoding apparatus, by determining
the first decoding algorithm and the second decoding algorithm such
that they correspond to the first coding algorithm and the second
coding algorithm respectively, and by determining the degree of
increasing the variation of the density levels such that it
correspond to the degree of decreasing the variation of the density
levels for obtaining the edge smoothed image, the coded information
obtained by the image coding apparatus can be properly decoded.
[0085] The density information correction part may include:
[0086] a mean value calculation part for calculating a mean value
of the density levels in a predetermined region; and
[0087] a density level judgement part for judging whether the
density level of a pixel is higher or lower than the mean value for
each pixel in the near region;
[0088] wherein density information is corrected for a pixel in
which the density level is higher than the mean value such that the
density level is increased, and density information is corrected
for a pixel in which the density level is lower than the mean value
such that the density level is lowered.
[0089] From the viewpoint that the average density level is not
changed when decoding, the density information correction part may
correct density information of each pixel in the near region such
that the mean value of the density levels does not change.
[0090] From the viewpoint of providing a concrete method of
sharpening the edge smoothed image, the edge sharpening part may
include:
[0091] a sharpening matrix generation part for generating, for each
block the edge part or a near region of the edge part in the edge
smoothed image, a sharpening matrix which is used for matrix
operation with a block density information vector, wherein the
block density information vector is obtained by arranging density
information of each pixel included in a block, and order of the
block density information vector corresponds to the number of
pixels in the block, and wherein the sharpening matrix includes
coefficients used for edge sharpening which operate on density
information of each pixel in the edge part or in a near region of
the edge part in the block; and
[0092] a matrix operation part for obtaining sharpened density
information of each pixel in the block by performing the matrix
operation by using the sharpening matrix and the block density
information vector.
[0093] According to the image decoding apparatus, the density
information of each pixel in the block can be obtained in the edge
sharpened image by performing matrix operation using the block
density information vector and the sharpening square matrix.
[0094] The image decoding apparatus can be configured such that it
is suitable for decoding the coded information obtained by the
before-mentioned image coding apparatus. In such case, from the
viewpoint of performing sharpening operation easily, the sharpening
matrix generation part may generate an inverse matrix of a
smoothing matrix as the sharpening matrix in which the smoothing
matrix is used for obtaining density information of the edge
smoothed image which is decoded from the coded density
information.
[0095] According to the image decoding apparatus, the sharpening
matrix used for sharpening the edge part of the edge smoothed image
can be obtained by obtaining an inverse matrix of the smoothing
matrix used for smoothing the original image.
[0096] From the viewpoint of providing a concrete method for
sharpening the edge smoothed image, the edge sharpening part may
include:
[0097] a pixel judgement part for judging whether a pixel exists in
the edge part represented by the edge information or in a near
region of the edge part while scanning the edge smoothed image
pixel by pixel;
[0098] a matrix operation part for performing, when the pixel
exists in the edge part or in the near region, matrix operation by
using a block density information vector and a sharpening matrix,
wherein the block density information vector is obtained by
arranging density information of each pixel included in a block,
the block includes the pixel and pixels in a surrounding region
around the pixel, and order of the block density information vector
corresponds to the number of pixels in the block, and wherein the
sharpening matrix includes coefficients used for edge sharpening
which operate on density information of each pixel in the
block;
[0099] an operation part for obtaining sharpened density
information of each pixel by overlaying density information of each
pixel in the block obtained by performing the matrix operation on
each pixel while scanning the edge smoothed image pixel by
pixel.
[0100] According to the image decoding apparatus, the density
information of each pixel in the block in the edge sharpened image
can be obtained by matrix operation using the block density
information block and the sharpening matrix while scanning the edge
smoothed image pixel by pixel.
[0101] The image decoding apparatus can be configured such that it
is suitable for decoding the coded information obtained by the
before-mentioned image coding apparatus. In such case, from the
viewpoint of performing sharpening operation easily, the sharpening
matrix may be an inverse matrix of a smoothing matrix in which the
smoothing matrix is used for obtaining density information of the
edge smoothed image which is decoded from the coded density
information.
[0102] According to the image decoding apparatus, the sharpening
matrix used for sharpening the edge part of the edge smoothed image
can be obtained as an inverse matrix of the smoothing matrix used
for smoothing the original image.
[0103] The second object can be also achieved by an image decoding
apparatus which decodes coded information of an image, the image
decoding apparatus including:
[0104] an edge information obtaining part for obtaining edge
information representing an edge part of the image;
[0105] a decoding part for obtaining density information of the
image by decoding the coded information according to a
predetermined decoding algorithm;
[0106] an edge sharpening part for sharpening the edge part
represented by the edge information for the density information of
the image such that a reproduced image is obtained.
[0107] The edge information obtaining part may obtain the edge
information by receiving it from the outside directly or by
decoding coded edge information.
[0108] In the latter case, the edge information obtaining part may
include an edge decoding part for obtaining the edge information by
decoding coded edge information which is provided to the image
decoding apparatus according to a predetermined decoding
algorithm.
BRIEF DESCRIPTION OF THE DRAWINGS
[0109] Other objects, features and advantages of the present
invention will become more apparent from the following detailed
description when read in conjunction with the accompanying drawings
in which:
[0110] FIG. 1 is a figure showing a principle of processes of an
image coding apparatus according to an embodiment of the present
invention;
[0111] FIG. 2 is a figure showing principle of processes performed
by an image decoding apparatus of an embodiment of the present
invention;
[0112] FIG. 3 is a block diagram of an image transmission system
including an image coding apparatus and an image decoding apparatus
according to an embodiment of the present invention;
[0113] FIG. 4 shows an example of a distance map;
[0114] FIG. 5 shows an example of an edge near region R;
[0115] FIG. 6 is a flowchart showing an example of a smoothing
process in the image coding apparatus;
[0116] FIG. 7 shows an example of a smoothing filter;
[0117] FIGS. 8A-8B shows each example of a smoothing filter matrix,
a unit matrix and a matrix for weight operation;
[0118] FIG. 9 shows an example of a sharpening filter matrix;
[0119] FIG. 10 shows an example of a method of correcting luminance
in a smoothing process;
[0120] FIG. 11 shows an example of a smoothing filter;
[0121] FIG. 12 shows another example of a smoothing filter
matrix;
[0122] FIG. 13 shows an example of a mask region used for the
smoothing process;
[0123] FIG. 14 is a flowchart showing an example of a process for
restoring (sharpening) an edge part of the edge smoothed image;
[0124] FIG. 15 is a block diagram of other configuration example of
an image transmission system including an image coding apparatus
and an image decoding apparatus;
[0125] FIG. 16 is a block diagram of still other configuration
example of an image transmission system including an image coding
apparatus and an image decoding apparatus.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0126] In the following, embodiments of the present invention will
be described with reference to figures.
[0127] The principle of processes performed by an image coding
apparatus of an embodiment of the present invention is as shown in
FIG. 1 for example.
[0128] As shown FIG. 1, edge information A is extracted from an
original image I.sub.O which is represented by density information
O (for example, luminance level) for each pixel, in which the edge
information A represents the edge part I.sub.E of the original
image I.sub.C. The extracted edge information A is coded according
to a standard coding algorithm such as JBIG and the like so that
coded edge information A.sub.C is obtained. From the extracted edge
information A, processes for smoothing the edge part I.sub.E are
performed on the density information representing the original
image I.sub.O so that density information B of an edge smoothed
image I.sub.1 is obtained for each pixel. The density information B
of the edge smoothed image I.sub.1 is coded according to the
standard coding algorithm such as JPEG so that coded density
information B.sub.C is obtained. This image coding apparatus
outputs the coded edge information A.sub.C and the coded density
information B.sub.C as coded information.
[0129] According to this image coding apparatus, the edge part
I.sub.E of the original image I.sub.O is smoothed for obtaining the
edge smoothed image I.sub.1 and the edge smoothed image I.sub.1 is
coded. Since the edge smoothed image I.sub.1 has relatively small
high spatial frequency components, information which is lost in
coding can be decreased even when the standard coding algorithm
using the DCT such as JPEG is used. In addition, when the edge
smoothed image I.sub.1 is coded by using DCT, efficiency of coding
by DCT around the edge of the edge smoothed image I.sub.1 can be
improved. Thus, if the edge part I.sub.E is coded individually,
degradation of coding efficiency can be prevented.
[0130] In addition, the principle of processes performed by an
image decoding apparatus of an embodiment of the present invention
is as shown in FIG. 2 for example.
[0131] In FIG. 2, the coded edge information Ac is decoded
according to an standard decoding algorithm such as JBIG
corresponding to the above-mentioned standard coding algorithm
(JBIG, for example) so that edge information A representing the
edge part I.sub.E of the original image is obtained. In addition,
the coded density information B.sub.C is decoded according to an
standard decoding algorithm such as JBEG corresponding to the
above-mentioned standard coding algorithm (JPEG, for example) so
that density information B' of the edge smoothed image I'.sub.1 is
obtained. Then, a sharpening process for sharpening the edge of the
edge smoothed image II.sub.1 is performed on the density
information B' of the edge smoothed image I'.sub.1 by using the
edge information A. Then, an image represented by density
information O' of each pixel is obtained as a reproduced image
I'.sub.O corresponding to the original image I.sub.O.
[0132] As mentioned above, in the image decoding apparatus, the
coded edge information A.sub.C and the coded density information B
obtained as the coded information are decoded, in which the coded
density information B.sub.C corresponds to the edge smoothed image.
By decoding the coded edge information A, the edge information
corresponding to the edge part of the original image I.sub.O can be
reproduced. Then, the edge part of the edge smoothed image I'.sub.1
which is obtained by decoding the coded density information B.sub.C
is sharpened by using the edge information obtained in the
above-mentioned way so that the reproduced image I.sub.O can be
obtained.
[0133] As for the edge smoothed image I.sub.1 obtained in the image
coding apparatus, although spatial resolution around the edge is
degraded, intrinsic visual information in the image signal (density
information) is not largely lost. Therefore, even when the
sharpening process for the edge of the edge smoothed image I.sub.1
is not performed, an image having a certain quality can be
obtained. Thus, the edge information can be used as enhanced
information of the original image. For example, in a system in
which coded information is transmitted from the image coding
apparatus to the image decoding apparatus, when information can not
be transmitted sufficiently due to network congestion, only the
coded density information of the edge smoothed image I.sub.1 may be
transmitted as the coded information. Accordingly, the image
transmission system can be configured such that the edge
information is used as necessary.
[0134] In this case, whether the coded edge information is
transmitted or not can be determined by negotiation between the
image coding apparatus and the image decoding apparatus. In
addition, by assigning priorities to the coded edge information and
the coded information of the edge smoothed image and by
IP-packeting these information, it becomes possible to transmit
high priority information to a decoding side accurately in internet
routers.
[0135] Next, an example which includes the image coding apparatus
and the image decoding apparatus of an embodiment of the present
invention will be described.
[0136] This image transmission system is configured as shown in
FIG. 3 for example. In this example, the image coding apparatus 10
and the image decoding apparatus 20 perform compression and
extension of a still image by using a standardized still image
coding method and decoding method such as JPEG and the like. The
image coding apparatus 10 includes functions for heightening
compression efficiency for regions around step edges such as
subject edge and character region and for separating edge
information from the original image and sending the edge
information hierarchically. According to these functions,
subjective quality around the edge of the image obtained by
decoding by the image decoding apparatus 20 can be improved.
[0137] In FIG. 3, in the image transmission system, the coded
information (coded edge information and coded density information)
of the original image is transmitted from the image coding
apparatus 10 to the image decoding apparatus 20 via a transmission
line 30. Then, the image decoding apparatus 20 decodes the coded
information so that the reproduced image is obtained.
[0138] The image coding apparatus 10 includes an edge detection
part 11, a JBIG coding part 12, a distance conversion part 13, an
edge smoothing part 14 and a JPEG coding part 15. The edge
detection part 11 detects an edge part from an input image
(original image), and outputs edge information representing the
edge part. A conventionally known method can be applied as a
detection method in the edge detection part 11. For example, a
generic method such as operators of Sobel can be used, in addition,
a process for optimizing detection accuracy of edges to be smoothed
can be applied in consideration of continuity of edge. The
information output from the edge detection part 11 is binary
information of white and black.
[0139] The JBIG coding part 12 performs undistortion (reversible)
compression on the edge information which is binary information and
output from the edge detection part 11 according to the JBIG coding
method. In addition, the edge information is converted into a
distance map by the distance conversion part 13. The distance
conversion process performed by the distance conversion part 13 is
a process for obtaining distribution of each distance between a
position (i,j) and a position (m,n) on the image. For example,
distance between an pixel of the edge part to the adjacent pixel is
represented by distance d(f.sub.ij, f.sub.mn) which is defined by
Euclidean distance, 4 adjacent distance, 8 adjacent distance and
the like. The distance map indicates a state of distance values
from the pixels of the edge part to the adjacent pixels. For
example, when the distance value is defined by the 4 adjacent
distance (d(f.sub.ij, f.sub.mn)=|i-m|+|j-n|), a distance map shown
in FIG. 4 can be obtained. In FIG. 4, each rectangular region
represents a pixel and the diagonally shaded rectangular region are
pixels of the edge part.
[0140] The edge smoothing part 14 receives the distance map (FIG.
4, for example) from the distance conversion part 13 and receives
the input image (original image), and performs smoothing process
for the part close to the edge in the image so that the edge
smoothed image is obtained. The details of the smoothing process
will be described later. According to this smoothing process, the
edge part including step edge and the like where the density
changes abruptly is smoothed.
[0141] The JPEG coding part 15 encodes the density information
representing the edge smoothed image received from the edge
smoothing part 14 according to JPEG. In this coding process, DCT
and quantization are performed. Since these processes are performed
on the edge smoothed image, image quality degradation which is
visually conspicuous such as block distortion, mosquito noise and
the like in the decoded image due to quantization of the DCT
coefficients can be suppressed, and a code amount for coding can be
decreased.
[0142] The image coding apparatus 10 outputs the coded edge
information from the JBIG coding part 12 and JPEG codes which is
the coded density information corresponding to the coded edge
information from the JBIG coding part 12 as the coded information.
At this time, the coded information and synchronization information
for identifying the same frames are multiplexed. The multiplexed
information is transmitted from the image coding apparatus 10 to
the image decoding apparatus 20 via the transmission line 30.
[0143] In the following, the process in the edge smoothing part 14
will be described concretely.
[0144] In this process, a smoothing filter matrix having an inverse
matrix is applied for pixels included in a region near edge (edge
near region) defined by the distance map (FIG. 4). In matrix
operation, a method of overlapping operation is used for
suppressing visual effects in boundaries of pixel blocks on which
the smoothing filter matrix operates.
[0145] First, assuming that a sample (pixel) of a pixel position
(i,j) of the input image is p.sub.ij and a distance value of the
pixel position (i,j) on the distance map (FIG. 4) is d(p.sub.ij).
In addition, an edge near region R is defined as
R={p.sub.ij|d(p.sub.ij)<.epsilon.} as shown in FIG. 5. The
smoothing process is performed to every p.sub.ij in the edge near
region R.
[0146] The smoothing process is performed according to a procedure
shown in FIG. 6 for example.
[0147] As shown in FIG. 6, the pixel position (i,j) is initialized
(i=0, j=0) and all values in an output image buffer are initialized
to 0 in step S1. In this state, it is judged whether the pixel
p.sub.ij is included in the edge near region R while scanning the
input image pixel by pixel (incrementing i, j) in step S2. More
particularly, it is judged whether the distance value d(p.sub.ij)
of the pixel satisfies d(p.sub.ij)<.epsilon.. When it is judged
that the pixel p.sub.ij is included in the edge near region R in
step S2 (YES), an matrix operation v'.sub.ij=OEv.sub.ij is
performed in step S3-1 wherein v.sub.ij represents a ninth order
pixel block vector V.sub.ij={I(p.sub.i-1, j-1),I(p.sub.i-1, . . . ,
I(p.sub.i+1 j+1)} obtained by arranging image values I (density
information) of nine pixels in a 3.times.3 pixel block including
the pixel p.sub.i j as the center, and E represents a ninth order
smoothing filter matrix which is regular and defined for
smoothing.
[0148] The smoothing filter matrix E is arranged such that each
coefficient in the smoothing filter shown in FIG. 7 is operated to
each pixel value included in the pixel block vector v.sub.ij by the
matrix operation (Ev.sub.ij). That is, the smoothing filter E is a
regular matrix of 9.times.9 order shown in FIG. 8A. Although the
9.times.9 order regular matrix is used for the smoothing filter
matrix E, it is not limited to this. The order of E can take any
value as long as the value is the same as the order of the pixel
block vector v.sub.ij which is the unit of the matrix operation. In
addition, the coefficients can be set according to the needed
smoothing operation. Accordingly, the smoothing filter matrix E can
be defined freely on condition that the inverse matrix used for
after-mentioned decoding exists.
[0149] To perform matrix operation between the smoothing matrix E
in which coefficients of the smoothing filter (FIG. 7) are arranged
and the pixel block vector v.sub.ij is equivalent to that the
smoothing filter (FIG. 7) is operated on the pixel p.sub.ij
included in the 3.times.3 pixel block as the center. Therefore,
according to the matrix operation, smoothing for the pixel p.sub.ij
is performed.
[0150] On the other hand, when it is judged that the pixel p.sub.ij
is not included in the edge near region in step 2 (NO), an
operation v'.sub.ij=OPv.sub.ij using the pixel block vector
v.sub.ij and a ninth order unit matrix P shown in FIG. 8B is
performed in step S3-2. When the unit matrix P is operated on the
image block vector v.sub.ij, the pixel block vector v.sub.ij is not
changed (Pv.sub.ij=v.sub.ij). Therefore, in this case, the
smoothing process is not performed.
[0151] The matrix O used for the matrix operations v'=OEv.sub.ij
and v'.sub.ij=OPv.sub.ij is configured as shown in FIG. 8C. Since
the above-mentioned operation is performed by shifting the pixel
p.sub.ij one by one, the smoothing process is performed while the
smoothing filter (FIG. 7) corresponding to the 3.times.3 pixel
block is overlapped (while each pixel becomes the center pixel of
the 3.times.3 pixel block or becomes a surrounding pixel). The
matrix O is for providing weight assigned in the overlapping
operation.
[0152] An operation result v'.sub.ij={I(p.sub.i-1, j-1),
I'(p.sub.i,j-1), . . . , I'(p.sub.i+1, j+1)} obtained by the matrix
operation represents each pixel value I' after performing smoothing
process for each pixel of the 3.times.3 pixel block. The pixel
value I'(p.sub.mn) is added on the output image buffer for each
pixel p.sub.mn in step S4.
[0153] After that, i and j are incremented for shifting the sample
pixel in step S5. In addition, it is judged whether each of i and j
exceeds S indicating the last pixel of the input image in step S6.
When each of i and j does not exceed S, the above-mentioned steps
(S2, S3-1 or S3-2, S4, S5, S6) are performed repeatedly. As a
result, the smoothing process is performed for the pixels p.sub.ij
included in the edge near region R while scanning the input image
pixel by pixel. After that, when each of i and j exceeds S, the
smoothing process ends. The values I' of the pixels stored in the
output pixel buffer at the end time becomes the density information
(pixel value data) representing the edge smoothed image.
[0154] As shown in FIG. 3, the image decoding apparatus 20 includes
a JBIG decoding part 21, a JPEG decoding part 22, a distance
conversion part 23, and an edge sharpening part 24. The image
decoding apparatus 20 receives the coded edge information and the
JPEG code (coded density information) corresponding to the edge
smoothed image where they are separated, wherein the coded edge
information and the JPEG code (coded density information) are sent
from the image coding apparatus 10 as the coded information.
[0155] The JBIG decoding part 21 generates edge information by
decoding the received coded edge information according to the
decoding method of JBIG. The JPEG decoding part 22 decodes the
received JPEG codes according to the decoding method of JPEG so
that density information representing the edge smoothed image is
generated. The distance conversion part 23 generates a distance map
(FIG. 4) on the basis of the edge information from the JBIG
decoding part 21 in the same way performed by the distance
conversion part 13 in the image coding apparatus 10. Then, the edge
sharpening part 24 performs sharpening for the edge part in the
edge smoothed image from the JPEG decoding part 22 on the basis of
distance value of each pixel of the edge near region, that is, the
edge sharpening part 24 performs intensity restoration of the edge
part.
[0156] In the edge sharpening part 24, following edge sharpening
process is performed for example.
[0157] In the process, a sharpening filter matrix E.sup.-1 which is
an inverse matrix of the smoothing filter matrix E is operated on
the pixels included in the edge near region which is defined by the
distance map. The process procedure is similar to that shown in
FIG. 6. In step S3-2 in FIG. 6, instead of the smoothing filter
matrix E, the sharpening filter matrix E.sup.-1 configured as shown
in FIG. 9 is used for matrix operation.
[0158] That is, while scanning the edge smoothed image pixel by
pixel, a matrix operation v.sub.ij=OE.sup.-1v'.sub.ij is performed,
wherein a ninth order pixel block vector v'.sub.ij={I'(p.sub.b.i-1,
j-1), I'(p.sub.i,j-1), . . . , I'(p.sub.i+1, j+1)} and the
sharpening filter matrix E.sub.-1 are used. The ninth order pixel
block vector v'.sub.ij is obtained by arranging pixel values of
each pixel included in a 3.times.3 pixel block having a pixel
p.sub.ij as the center which is included in the edge near region.
As a result, sharpening is performed for the pixel p.sub.ij.
[0159] Then, at the time when the above-mentioned process ends for
every pixel in the edge smoothed image, pixel values of pixels
stored in the output image buffer becomes the density information
corresponding to the original image. The density information is
output to the image decoding apparatus 20 as output image
information.
[0160] According to the above-mentioned image transmission system,
the image coding apparatus 10 detects edge position of the original
image (input image) and performs the smoothing process on pixels
near the edge. Thus, loss of information at the time of coding due
to the edge are prevented, and degradation of coding efficiency can
be prevented. In addition, since the smoothing of the edge is
performed by the matrix operation using the smoothing filter matrix
which has an inverse matrix mathematically, sharpening of edge can
be performed easily by performing the inverse operation of the
smoothing process by the image decoding apparatus 20, that is,
intrinsic resolution of the edge region can be restored.
[0161] In addition, the output corresponding to the edge smoothed
image from the JPEG decoding 22 in the image decoding apparatus 20
can be used for displaying an image and the like as image data
(density information). The edge information can be used for
restoring edge intensity as necessary. This means that texture of
the edge part can be restored by adding an edge sharpening function
to a product which complies with the standard image coding method
which is widely distributed currently. Therefore, a system which
performs compression and extension of an image can be realized
without largely changing basic structure of an image encoding
apparatus which supports existing standard image coding method.
Since the amount of processing corresponding to the function to be
added for restoring edge intensity is smaller than that of the
compression coding processing such as JPEG and the like, the
function can be added to the image decoding apparatus as a software
module for example.
[0162] Next, other examples of the edge smoothing part 14 in the
image coding apparatus 10 and the edge sharpening part 24 in the
image decoding apparatus 20 will be described.
[0163] First, the edge smoothing part 14 divides the input image
(original image) into N.times.N pixel blocks B (for example,
8.times.8 pixel DCT blocks), and performs following processes for
each block B which includes the edge part of the original image.
The block to be processed may include a plurality of edge
parts.
[0164] Assuming that a pixel value (density information: luminance
(brightness) value) of a pixel k in the block B is
I.sub.k(k.epsilon.B). A mean value E.sub.B of the luminance values
I.sub.k of the pixels in the block B is calculated. Then, on the
basis of the distance value d.sub.k of each pixel of the distance
map (FIG. 4) generated in the distance conversion part 13, it is
judged whether the pixel k is within the edge near region
(d.sub.k<.epsilon.). When the pixel k is judged to be within the
edge near region R, the luminance value I.sub.k of the pixel k is
corrected according to the distance value d.sub.k.
[0165] The mean value E.sub.B is not limited for calculated on the
basis of the block B. The mean value E.sub.B can be also calculated
on the basis of luminance values of pixels in a predetermined
region including an edge part.
[0166] In the luminance correction, it is judged whether the
luminance value I.sub.k of the pixel k in the block B is higher or
lower than the mean value E. Then, as shown in FIG. 10, when a
luminance value of a pixel p.sub.2 is judged to be higher than the
mean value E.sub.B, the luminance value of the pixel p.sub.2 is
decreased (.dwnarw.) On the other hand, when a luminance value of a
pixel p.sub.0 is judged to be lower than the mean value E.sub.B,
the luminance value of the pixel p.sub.0 is increased (.uparw.).
Then, the correction (decrease, increase) of the luminance value is
performed such that a mean value E'.sub.B of luminance values
I'.sub.k of the pixels in the block B after correction (before
smoothing) becomes the same as the mean value E.sub.B, and such
that the correction (decrease, increase) is inversely proportional
to the distance value d.sub.k (distance from the edge part) of each
pixel k.
[0167] More particularly, a corrected luminance value I.sub.k of a
pixel is calculated according to the following procedure for each
pixel, wherein .alpha. is a constant which satisfies
0<.alpha.<1.
If (I.sub.k<E.sub.R) then
I'.sub.k=I.sub.k+(E.sub.R-I.sub.k)*(1-.alpha.d.sub.k)
else if (I.sub.k>E.sub.R) then
I'.sub.k=I.sub.k-(I.sub.k-E.sub.R)*(1-.alpha.d.sub.k)
[0168] According to this procedure, the luminance value I.sub.k of
the pixel k having distance d.sub.k in the edge near region R is
corrected such that the luminance value I.sub.k moves closer to the
mean value E.sub.k by a weighted difference between the means value
E.sub.R and the luminance value I.sub.k in which a weight is
assigned to the difference according to the distance.
[0169] According to this process, the luminance values of pixels
which are arranged across the edge part are corrected such that
variation of the luminance values of the pixels becomes small (that
is, such that the luminance value moves closer to the mean value
E.sub.B). Therefore, variation of luminance of the edge part
becomes small so that the edge part is smoothed.
[0170] In the edge sharpening part 24 in the image decoding
apparatus 20 which sharpens the edge smoothed image which is
obtained by smoothing according to the above-mentioned procedure,
processes are performed similarly to the smoothing processes.
[0171] The edge smoothed image obtained by the JPEG decoding part
22 is divided into N.times.N pixel blocks B like the
above-mentioned way and each of the blocks is processed. That is,
in a block B including an edge part, a luminance value I'.sub.k of
a pixel k in an edge near region is-corrected such that the
luminance value I'.sub.k is increased when the luminance value
I'.sub.k is larger than a mean value E'.sub.B of luminance values
of the block B, and the luminance value I'.sub.k is decreased when
the luminance value I'.sub.k is smaller than the mean value
E'.sub.B. More specifically, the luminance value I.sub.k of each
pixel after correction (after sharpening) is calculated by
I k = I k ' - E R ' * ( 1 - .alpha. k ) .alpha. k and E R ' = 1 N k
.di-elect cons. R I ' k . ##EQU00003##
[0172] According to the sharpening process, the luminance values of
pixels which are arranged across the edge part are corrected such
that variation of the luminance values becomes large (that is, such
that the luminance value moves away from the mean value E.sub.B) in
the edge near region in the edge smoothed image. Therefore,
variation of luminance of the edge part in the edge smoothed image
becomes large so that the edge part is sharpened (edge intensity is
restored).
[0173] In the next example, the edge smoothing part 14 in the image
coding apparatus 10 divides an input image (original image) into
8.times.8 pixel blocks B (DCT blocks), and performs following
processes for each block B including the edge part of the original
image. The block B to be processed may include a plurality of edge
parts.
[0174] A smoothing filter matrix A which is a 64.times.64 order
square matrix is generated on the basis of the edge information
from the edge detection part 11 and the distance map from the
distance conversion part 13, wherein the smoothing filter matrix A
is operated on a 64 order pixel block vector v which is obtained by
arranging pixel values I (density information) of pixels (64
pixels) included in the block B (8.times.8 pixel block) which
includes the edge part. The configuration of the smoothing filter
matrix A depends on the position of the edge included in the block
B.
[0175] In the smoothing filter matrix A, coefficients are arranged
such that matrix operation between the matrix A and the pixel
block-vector v is equivalent to that smoothing process by a
smoothing filter of 3.times.3 pixel block (FIG. 11 for example) is
performed on a pixel in an edge part or in the near region (in
diagonally shaded area in FIG. 4 for example) in the 8.times.8
pixel block.
[0176] More particularly, the smoothing filter matrix A is
configured as shown in FIG. 12. The configuration of the smoothing
filter matrix A is for a case where only a pixel of a position
(m,n) is included in the edge part or the near region.
[0177] After the smoothing filter matrix A is obtained on the basis
of the edge information and the distance map, the matrix operation
v'=Av is performed wherein the pixel block vector v corresponding
to the block B and the smoothing filter matrix A are used. Each
element of a vector v' (corrected pixel value) obtained by the
matrix operation becomes a pixel value of each pixel in a block
after smoothed is performed. Accordingly, the smoothing process is
performed in the block.
[0178] The configuration of the smoothing filter matrix A (FIG. 12)
and the configuration of the considered smoothing filter (FIG. 11)
are not limited to the above-mentioned examples. These may vary as
necessary.
[0179] In the edge sharpening part 24 in the image decoding
apparatus 20 which performs sharpening of the edge smoothed image
which is obtained by the smoothing process according to the
above-mentioned procedure, processes are performed according to the
similar procedure to the above-mentioned smoothing process. For the
sharpening process, a sharpening filter matrix A.sup.-1 which is an
inverse matrix of the smoothing filter matrix A is generated for
each block (8.times.8 pixel block) on the basis of the edge part
and the near region in the block which are defined by the edge
information and the distance map. Then, pixel values of an edge
sharpened image (where edge intensity is restored) are obtained by
a matrix operation v=A.sup.-1v' for each of blocks of the edge
smoothed image, wherein the matrix operation uses a pixel block
vector v' configured by pixel values of pixels included in each
block and the sharpening filter matrix A.sup.-.
[0180] In such image coding/decoding method, it is necessary to
generate the smoothing filter matrices A and the inverse matrices
A.sup.-1 corresponding to edge positions for each block. However, a
filter which has eighth order tap at the maximum vertically and
horizontally can be configured, desired smoothing characteristics
can be obtained.
[0181] In the next example, the edge smoothing part 14 in the image
coding apparatus 10 calculates smoothed pixel values x' of each
pixel p by performing following processes on pixel values x
(density information) of each pixel p (p.epsilon.R) which are
judged to be within the edge near region (FIG. 5) in the input
image (original image).
[0182] For example, as shown in FIG. 13, a mask region of 3
pixels.times.3 pixels is defined where the pixel p is centered, and
a mean value C of pixel values x.sub.1-x.sub.8 of eight pixels
p.sub.1-p.sub.8 around a pixel p in the mask region. Then, a pixel
value x' of the pixel p which is smoothed is obtained by
x'=(1-.lamda.)x+.lamda.C (equation 1) in consideration of the mean
value of the pixel values x.sub.1-x.sub.8 the surrounding pixels.
In the equation, .lamda. is a positive constant equal to or smaller
than 0.5 and can take any value within this range. As the value
.lamda. becomes greater, ratio of the pixel value (mean value C) of
surrounding pixels which is reflected to the pixel value x' becomes
larger so that the degree of smoothing becomes greater. Although
the mean value of the pixel values of the eight pixels
p.sub.1-p.sub.8 around the pixel p is used as the value of C in
this example, C can be determined by using other methods as long as
density state of surrounding pixels is reflected. In this case, the
value of A can be determined as a positive number according to the
method of determining C.
[0183] In the image decoding apparatus 20, the edge sharpening part
24 sharpens the edge part of the edge smoothed image (the edge
smoothed image output from the JPEG decoding part 22) obtained by
the above-mentioned smoothing process. The sharpening process is
performed in the following way for example.
[0184] Original pixel values x are restored, in which the
relationship between the original pixel values x and the pixel
values x' of pixels of the edge near region R is represented by the
equation 1, wherein the pixel values x' are obtained from the edge
information from the distance conversion part 23. In this case,
since the subject to be processed is the edge smoothed image, the
same value as the mean value C used for the smoothing process can
not be used as a mean value C of the pixel values of the
surrounding pixels in this process, wherein the input image
(original image) is processed in the smoothing process. Therefore,
the original image values x are restored by repeating operation
complying with the steepest-descent method. When assuming that the
restored pixel value is {circumflex over (x)}, since
^ + 1 1 - .lamda. ( .lamda. C - x ' ) = 0 ##EQU00004##
is satisfied from the equation 1,
e ( X ) = ( X + 1 1 - .lamda. ( .lamda. C ( n ) - x ' ) ) 2 (
equation 2 ) ##EQU00005##
is defined. Then, the restored pixel value is defined as X which
minimizes e(X). Since C is changed depending on calculation process
of X, C is described such that the value depends on the number of
repetition n.
[0185] According to the steepest-descent method, X which minimizes
e(X) can be obtained by a convergence value of a following
recurrence equation,
X ( n + 1 ) = X ( n ) - G * .differential. e .differential. X ' , .
( equation 3 ) ##EQU00006##
In this equation, G is a minute constant and is determined as an
optimum value on the basis of the state of actual image represented
by restored values of pixels.
[0186] The process for obtaining the restored pixel value of each
pixel by the repeating operation by the equation 3 is performed
according to a procedure shown in FIG. 14 for example.
[0187] In FIG. 14, first, an output buffer is initialized in step
S11 and an initial value is set to each variable in step S12. That
is, initial value "0" is set to a repetition count number n (n=0)
and the pixel value x' of the edge smoothed image is set to the
initial value X.sub.ij(O) of a restored image value X.sub.ij of a
pixel p.sub.ij for each pixel. In addition, "0" is set to each flag
Flag(i,j) which corresponds to each pixel p.sub.ij (Flag(i,j)=0).
The flag Flag(i,j) indicates whether the restored pixel value of
the pixel p.sub.ij is converged by the after-mentioned repeating
operation.
[0188] In this state, a mean vale C.sub.ij(n) of pixel values of
eight pixels which surround the pixel p.sub.ij in the edge smoothed
image is calculated in step S13, and a differential value of e(X)
in the equation 2 is calculated by
.differential. e .differential. X ( X = Xij ( n ) ) ( equation 4 )
##EQU00007##
in step S14. Then, it is judged whether the differential value
satisfies a condition
.differential. e .differential. X ( n ) < ( : constant ) (
equation 5 ) ##EQU00008##
(equation 5) in step S15.
[0189] If this condition is not satisfied (N in step S15), that is,
if it is not regarded that the restored pixel value of the pixel
p.sub.ij is converged to a value, X.sub.ij(n+1) is calculated
according to the equation 3 by using C.sub.ij(n) and the
differential value in step S16. When the restored image value
X.sub.ij(n+1) is thus obtained, the restored pixel value
X.sub.ij(n) of the pixel p.sub.ij in the output buffer is updated
into X.sub.ij(n+1) in step S17.
[0190] After that, it is judged whether the process is performed
for all pixels in the edge near region R in the edge smoothed image
in step S18. When the process is not completed for all pixels (N in
step S18), (i,j) is incremented in step S19 so that the pixel to be
processed is changed.
[0191] When the pixel to be processed is changed, it is judged
whether the flag Flag(i,j) is "0" in step S20. When this flag
Flag(i,j) is "0", it means that the restored pixel value of the
pixel is not converged. Thus, the mean value C(n) of pixel values
(restored pixel values) of eight pixels surrounding the pixel is
calculated in step S13 and the differential value is calculated
according to the above-mentioned equation 4 in step S14. Then, it
is judged whether the condition represented by the equation 5 is
satisfied, that is, whether it can be regarded that the restored
pixel value is converged in step S15. When this condition is not
satisfied (N in step S15), in the same way as the above-mentioned
process, X.sub.ij(n+1) is calculated according to the
above-mentioned equation 3 in step S16, and the restored pixel
value of the pixel p.sub.ij in the output buffer is updated to
X.sub.ij(n+1) in step S17.
[0192] Similar processes (S18, S19, S20, S13-S17) are repeated
hereinafter. Then, when it is judged that the above-mentioned
process is completed for all pixels in the edge near region R in
the edge smoothed image (Y in S18), it is judged whether each flag
Flag(i,j) of the pixels is "1", that is, whether the restored image
value can be regarded to be converged for every pixel in step
S21.
[0193] When it is not judged that the flag Flag(i,j) is "1" for all
pixels (N in step S21), the repetition count number n is
incremented, and the initial value (0,0) is set to (i,j) so that
the pixel p.sub.00 which was processed first is again selected as a
pixel to be processed in step S22. Then, it is judged whether the
flag Flag(0,0) corresponding to the pixel p.sub.00 is "0" in step
S20. When the flag Flag(0,0) is "0", the restored pixel value of
the pixel p.sub.00 is updated according to the above-mentioned
processes (S13-S17). After that, the above-mentioned processes
(S20, S13-S19, S21) are performed repeatedly while changing the
pixel to be processed one by one.
[0194] In this process, if the condition represented by the
equation 5 is satisfied (Y in S15), it can be regarded that the
restored pixel value X.sub.ij(n) corresponding to the pixel
p.sub.ij stored in the output buffer is converged, and "1" is set
to the flag Flag(i,j) of the pixel p.sub.ij in step S23. The
above-mentioned processes (S18-S20, S13-S17, S18, S21, S22) are
repeated hereinafter. In these processes, every time when the
condition represented by the equation 5 is satisfied (Y in S15),
"1" is set to the flag Flag(i,j) corresponding to the pixel
p.sub.ij.
[0195] When it is judged that the Flag(i,j) is "1" for every pixel
to be processed (Y in S21), that is, when it is regarded that the
restored pixel value is converged for every pixel to be processed,
the sharpening process of the edge part of the edge smoothed image
is completed. At the time, the restored pixel values X for all
pixels are stored in the output buffer.
[0196] According to the smoothing process for the edge part of the
original image and the sharpening process of the edge part of the
edge smoothed image which is obtained by the smoothing process, the
restored pixel values can be obtained with relatively high accuracy
by performing the above-mentioned repeating operation regardless of
degree of sharpening of each pixel according to the equation 1. In
addition, by properly setting parameters in the edge restoring
process (edge sharpening process), the number of repetition of the
repeating process can be decreased so that the process time can be
decreased.
[0197] Next, other configuration example of the image coding
apparatus 10 and the image decoding apparatus 20 will be
described.
[0198] In an image transmission system shown in FIG. 15, coding and
decoding of moving images are performed. Therefore, an MPEG-4 video
coding part 16 and an MPEG-4 video decoding part 25 are used
instead of the JPEG coding part 15 in the image coding apparatus 10
and the JPEG decoding part 22 in the image decoding apparatus 20 in
FIG. 3.
[0199] In the image coding apparatus 10, a frame image of an edge
smoothed image obtained by the edge smoothing process for the input
image is video-compressed and coded according to the standard
method of MPEG-4 in the MPEG-4 video coding part 16. As a result,
MPEG-4 code can be obtained as coded information.
[0200] In the image decoding apparatus 20, the MPEG-4 code is
decoded for each frame image by the MPEG-4 video decoding part 25.
Then, the edge sharpening part 24 performs edge sharpening process
on each frame decoded image by using corresponding edge
information. Accordingly, a moving image corresponding to an
original image (moving image) is output from the image decoding
apparatus 20.
[0201] In an image transmission system shown in FIG. 16, coding and
decoding of a moving image are performed. In this image
transmission system, an MPEG-4 shape coding part 17 and an MPEG-4
shape decoding part 26 are used instead of the JBIG coding part 12
which performs coding of edge information in the image coding
apparatus 10 and the JBIG decoding part 21 which decodes the coded
edge information in the image decoding apparatus 20 shown in FIG.
3.
[0202] In the image coding apparatus 10, the edge information is
compressed and coded without distortion by the MPEG-4 shape coding
part 17. Accordingly, in addition that the edge image is coded
within each closed frame, the edge image can be compressed by using
a correlation between frames in the direction of time. Thus,
compression efficiency of the edge information can be improved. In
addition, the MPEG-4 shape coding part 17 can output code of only
binary image data (MPEG-4 shape code) as the coded edge information
by using BINARY ONLY coding mode of the MPEG-4 core profile. Then,
the image decoding apparatus 20 which performs desired operation
can be configured only by providing a decoding part complying with
the MPEG-4 core profile (MPEG-4 shape decoding part 26 and MPEG-4
video decoding part 25), and, providing a function for performing
processes on the MPEG-4 shape code (used for edge sharpening after
distance conversion).
[0203] In each of the above-mentioned examples, although the image
decoding apparatus 20 decodes coded edge information and coded
density information obtained by coding density information of the
edge smoothed image sent from the image coding apparatus 10, each
coded information may be provided to the image decoding apparatus
20 by using a recording medium such as MO (magneto-optical
disk).
[0204] In addition, the coded density information which is obtained
by coding density information of the edge smoothed image and
uncoded edge information may be provided to the image decoding
apparatus 20. In this case, the image decoding apparatus 20 obtains
a reproduced image by performing the sharpening process on density
information on the basis of edge information which is obtained by
means of some kind (receiving from transmission line, reading out
from recording medium and the like), wherein the density
information is obtained by decoding the coded density
information.
[0205] As mentioned above, according to the present invention,
relatively high spatial frequency components decrease in the edge
smoothed image obtained by smoothing the edge part of the original
image. Thus, even when the image is coded according to a standard
coding algorithm, information which is lost by coding decreases. As
a result, an image coding/decoding method for obtaining a decoded
image of low distortion even when the standard coding/decoding
method using DCT is applied can be realized.
[0206] In addition, according to the present invention, an image
coding apparatus and an image decoding apparatus which perform
coding and decoding of an image according to the image
coding/decoding method can be realized.
[0207] The present invention is not limited to the specifically
disclosed embodiments, and variations and modifications may be made
without departing from the scope of the invention.
* * * * *