U.S. patent application number 10/484537 was filed with the patent office on 2005-01-06 for error concealment for image information.
Invention is credited to Hartung, Franck, Schuba, Marko.
Application Number | 20050002652 10/484537 |
Document ID | / |
Family ID | 8177993 |
Filed Date | 2005-01-06 |
United States Patent
Application |
20050002652 |
Kind Code |
A1 |
Hartung, Franck ; et
al. |
January 6, 2005 |
Error concealment for image information
Abstract
Method and apparatus for error concealment in an image allowing
to conceal an error area in the image with estimated pixels. A
first boundary section and a second boundary section adjacent to
the error area is determined and correspondences between the
boundary elements of the boundary sections are established using
non-linear alignment operations. After establishing the
correspondences pixels between respective boundary elements of the
first boundary section and the second boundary section are
estimated. The non-linear alignment operations may include dynamic
programming techniques, including Needleman-Wunsch techniques,
wherein a similarity measure is used for a matrix fill
operation.
Inventors: |
Hartung, Franck;
(Herzogenrath, DE) ; Schuba, Marko; (Aldenhoven,
DE) |
Correspondence
Address: |
ERICSSON INC.
6300 LEGACY DRIVE
M/S EVR C11
PLANO
TX
75024
US
|
Family ID: |
8177993 |
Appl. No.: |
10/484537 |
Filed: |
September 8, 2004 |
PCT Filed: |
July 5, 2002 |
PCT NO: |
PCT/EP02/07526 |
Current U.S.
Class: |
386/263 ; 360/32;
375/E7.281 |
Current CPC
Class: |
H04N 19/895
20141101 |
Class at
Publication: |
386/114 ;
360/032 |
International
Class: |
H04N 007/64; G11B
005/00; G11B 005/09 |
Foreign Application Data
Date |
Code |
Application Number |
Jul 17, 2001 |
EP |
01116704.6 |
Claims
1. Method for error concealment in an image, comprising the steps
of: detecting an error area in image data of an image consisting of
pixels; determining a first boundary section and a second boundary
section of the error area, the boundary sections including boundary
elements being defined based on pixels with image information close
to the error area; aligning the boundary elements of the first
boundary section and the second boundary section using alignment
operations to establish correspondences between respective boundary
elements of the first boundary section and the second boundary
section; and estimating pixels of the error area based on the
established correspondences between respective boundary elements of
the first boundary section and the second boundary section.
2. Method of claim 1, wherein the alignment operations include
non-linear alignment operations including dynamic programming
techniques.
3. Method of claim 1, wherein the alignment operations include
Needleman-Wunsch techniques.
4. Method of claim 1, wherein a boundary element similarity measure
is used for a matrix fill operation.
5. Method of claim 1, wherein the boundary element similarity
measure includes classifying the boundary elements into a plurality
of ranges of pixel parameters, a pixel parameter being constituted
by at least one of grey level values and colour values.
6. Method of claim 5, wherein the width of the individual ranges
corresponds to a distribution of the parameter values.
7. Method of claim 1, wherein the first and second boundary section
each include an array of pixels adjacent to the error area and are
located at opposing sides of the error area.
8. Method of claim 1, wherein the boundary elements are each
determined based on at least one image pixel.
9. Method of claim 1, wherein the first and second boundary section
include different numbers of boundary elements.
10. Method of claim 1, wherein the establishing of correspondences
includes a correspondence from a boundary element of the first
boundary section to no boundary element or at least one boundary
element of the second boundary section.
11. Method of claim 1, wherein the error area is constituted by
missing parts of rows of the image information and the first and
second boundary section are constituted by row sections of rows of
error free pixels of the image.
12. Method of claim 1, wherein the pixels of the error area are
estimated using an interpolation technique.
13. Method of claim 1, wherein the image forms an image of a video
sequence of images transmitted in uncompressed or compressed form
over a packet network.
14. Method of claim 1, wherein the image is a first image of a
sequence of images and the estimated pixels are used for displaying
at least one second image of the sequence, the second image
following or preceding the first image.
15-17. (Cancelled)
18. Apparatus for error concealment in an image, comprising: a
detecting unit for detecting an error area in image data of an
image consisting of pixels; a determining unit for determining a
first boundary section and a second boundary section of the error
area, the boundary sections including boundary elements being
defined based on pixels with image information close to the error
area; an aligning unit for aligning the boundary elements of the
first boundary section and the second boundary section using
non-linear alignment operations to establish correspondences
between respective boundary elements of the first boundary section
and the second boundary section; and an estimating unit for
estimating pixels of the error area based on the established
correspondences between respective boundary elements of the first
boundary section and the second boundary section.
19. Apparatus of claim 18, wherein the alignment operations include
non-linear alignment operations including dynamic programming
techniques.
20. Apparatus of claim 18, wherein the alignment operations include
Needleman-Wunsch techniques.
21. Apparatus of claim 18, wherein a boundary element similarity
measure is used for a matrix fill operation.
22. Apparatus of claim 18, wherein the boundary element similarity
measure includes classifying the boundary elements into a plurality
of ranges of pixel parameters, a pixel parameter being constituted
by at least one of grey level values and colour values.
23. Apparatus of claim 22, wherein the width of the individual
ranges corresponds to a distribution of the parameter values.
24. Apparatus of claim 18, wherein the first and second boundary
section each include an array of pixels adjacent to the error area
and located at opposing sides of the error area.
25. Apparatus of claim 18, wherein the boundary elements are each
determined based on at least one image pixel.
26. Apparatus of claim 18, wherein the first and second boundary
section include different numbers of boundary elements.
27. Apparatus of claim 18, wherein the aligning unit establishes of
correspondences from a boundary element of the first boundary
section to no boundary element or at least one boundary element of
the second boundary section.
28. Apparatus of claim 18, wherein the error area is constituted by
missing parts of rows of the image information and the first and
second boundary section are constituted by row sections of rows of
error free pixels of the image.
29. Apparatus of claim 18, wherein the estimating unit is adapted
to estimate the pixels of the error area using an interpolation
technique.
30. Apparatus of claim 18, wherein the image forms an image of a
video sequence of images transmitted in uncompressed or compressed
form over a packet network.
31. Apparatus of claim 18, wherein the image is a first image of a
sequence of images and the estimated pixels are used for displaying
at least one second image of the sequence, the second image
following or preceding the first image.
Description
FIELD OF THE INVENTION
[0001] The present invention relates to a method and apparatus for
error concealment in an image, video or image transmission.
TECHNOLOGICAL BACKGROUND
[0002] With increased processing capabilities of data processing
devices, a growing number of applications makes use of
sophisticated graphics and/or video information in providing
services for a user. For example, an application may require the
display of a single image or a sequence of images on a display. In
this case the image information may be retrieved from a local or
remote source or storage unit, subjected to further processing, if
necessary, and then displayed on a display accessed by a data
processing device.
[0003] Digital representations of images generally consist of a
large number of pixels representing the image information, for
example grayscale information, color information and similar. When
displaying the image pixels on a display, they generate a
representation of the original image.
[0004] As images and particularly sequences of images of a video
stream may include large amounts of data, it may be desired to
compress the image information in order to reduce the amount of
data representing the image. For example, an image can be
compressed according to some compression algorithm and then stored
on a storage device, requiring less storage space as compared to
storing the image in an uncompressed format. If the image is then
to be displayed, a corresponding decompression algorithm can be
used to decompress the compressed image information prior to
displaying it on the display. Compressing the image information can
significantly reduce the required amount of information for
representing an image or a sequence of images, and thus allows to
save storage space.
[0005] Further, if a source of an image or image sequence is
located remote from a data processing device used for displaying
the image or sequence of images, compressing the image information
can lead to the further advantage of reducing a data rate
requirement for transmitting the image information from a storage
location or any other source of image information, such as a camera
or similar, to the device for actual display. Thus, not only
storage space requirements can be reduced, but also bandwidth
requirements for transmitting image or video information.
[0006] Several international standards for image/video compression
and/or for transmission of compressed video information over packet
networks like the Internet or mobile networks exist. An example
thereof is the MPEG-4 (Moving Pictures Experts Group) video
compression, combined with RTP(Realtime Transport
Protocol)/UDP(User Datagram Protocol) packetization and
transmission.
[0007] Compressed image or video information can be transmitted
between data processing devices over any kinds of networks or
communication links between data processing devices, including
packet switched networks. Packet switched networks are widely used
and include wide area networks such as the Internet or local area
networks such as company-wide intranets or mobile packet switched
networks and similar. Information to be transmitted in packet
switched transmission networks is generally divided into
information packets which are individually transmitted over the
network to a recipient. At the recipient the information included
in a plurality of packets is combined and further processed.
[0008] However, when transmitting packets over packet switched
networks, particularly when using unreliable protocols, typically
some packet loss occurs, i.e. some packets can be lost or delayed
beyond a threshold allowing processing during transmission. Thus,
parts of the information scheduled for transmission does not arrive
at the recipient, leading to incomplete data for further
processing. Typically, information loss when transmitting and/or
retrieving images or video sequences of images affects part of a
single or a plurality of images. For example, an information loss
may affect a region within an image to be displayed, which
therefore will be perceived as incomplete or distorted by a viewer
unless error concealment is applied.
[0009] Further, information loss or corruption can occur when
retrieving and/or transmitting image data in compressed or
uncompressed form due to various other reasons. For example in
packets with bit errors delivered for an application or in a bit
stream (circuit switched) with bit errors part of the image
information is still present in an area.
[0010] It is therefore desired to conceal the missing or corrupted
region of an image such that a viewer does not readily notice the
missing part of the image.
[0011] Several techniques for concealment in video and image
transmission exist. Some of these methods perform an error
concealment based on available information of the image, and thus
estimate and conceal the lost pixels only from the remaining pixels
of a picture. Such methods are called intra-frame concealment
methods. Others try to estimate and conceal the lost pixels from
pixels of previous and/or from the remaining pixels in the picture
and possibly also following frames of a video sequence. Such
methods are called inter-frame concealment methods, as for example
outlined in "Error control and concealment for video communication:
A review", Yao Wang and Qin-Fan Zhu, Proceedings of the IEEE, Vol.
86, No. 5, May 1998.
[0012] Intra-frame concealment methods may simply apply by linear
interpolation between corresponding pixels on two sides of a
missing region, more advanced methods use block-based interpolation
by looking for dominant edges and attempting to continue the edges.
Other methods try to interpolate the block-based coefficient
domain.
[0013] However, all of the above methods fail to provide good error
concealment in all possible situations, including characteristics
of the images to be handled.
SUMMARY OF THE INVENTION
[0014] It is therefore an object of the invention to provide for an
improved method for error concealment allowing a more flexible and
efficient error concealment in an image.
[0015] This object of the invention is solved by a method for error
concealment in an image, including: detecting an error area in
image data of an image consisting of pixels; determining a first
boundary section and a second boundary section of the error area,
the boundary sections including boundary elements being defined
based on pixels with image information close or adjacent to the
error area; aligning the boundary elements of the first boundary
section and the second boundary section using alignment operations
to establish correspondences between respective boundary elements
of the first boundary section and the second boundary section; and
estimating pixels of the error area based on the established
correspondences between respective boundary elements of the first
boundary section and the second boundary section.
[0016] Accordingly, the invention allows, by aligning the boundary
elements using alignment operations, to provide improved
correspondences between the boundary elements and thus allows to
estimate error concealment information between the established
correspondences more closely resembling missing or corrupted image
information. Accordingly, the concealment of errors in the image
can be improved, both in terms of similarity to the lost
information and in terms of visibility for an observer.
[0017] The alignment operations may include non-linear alignment
operations including dynamic programming techniques. The alignment
operations may include Needleman-Wunsch techniques. The alignment
operations may allow to establish correspondences between
respective boundary elements of the boundary sections at reduced
computational complexity.
[0018] Preferably, a similarity measure may be used in the
alignment operation. For example, a boundary element similarity
measure may be used for a matrix fill operation. By providing a
similarity measure, computational requirements can be further
reduced, as boundary elements having certain similar values can be
grouped together, for example into ranges of values not showing a
significant perceptual difference for a viewer. Further,
sensitivity to minor variations can be reduced, e.g., noise,
artifacts, illumination, variations in the object itself, etc.
[0019] The similarity measure may include classifying the boundary
elements into a plurality of ranges of pixel parameters, a pixel
parameter being constituted by at least one of gray level values
and color values. By providing a plurality of ranges for the pixel
parameters, a classification of the boundary elements using the
similarity measure can be effected at reduced computational
requirements.
[0020] Advantageously, the width of the individual ranges may
correspond to a distribution of the parameter values. This allows
to adapt the similarity measure, i.e., the ranges, to particular
image characteristics, for example if certain parameter values of
the boundary pixels are encountered less frequently, e.g. if the
boundary sections are generally dark or bright, the adapted width
of the ranges allows an improved establishment of correspondences
between boundary pixels.
[0021] The first and second boundary section may each include an
array of pixels lying adjacent to the error area located at
opposing sides of the error area. The arrays of pixels may be
either continuous or discontinuous.
[0022] Each of the boundary elements may be based on at least one
image pixel.
[0023] The first and second boundary sections may include different
numbers of boundary elements, allowing a more flexible definition
of boundary sections, for example depending on the size of the
error area.
[0024] The establishing of correspondences between boundary
elements may include establishing a correspondence from a boundary
element of the first boundary to no boundary element or at least
one boundary element of the second boundary section. Thus, a
particular boundary element may be defined to correspond to one or
a plurality of boundary elements or may be defined to not
correspond to any boundary element of the second boundary section,
allowing further flexibility in establishing the
correspondences.
[0025] The error area may be constituted by missing parts of rows
of the image information and the first and second boundary section
may be constituted by row sections of rows of error free pixels of
the image. Thus, particularly when applying the invention to
compressed image information where often blocks of row sections are
missing, e.g. due to packet switched transmission losses, the
boundary sections can directly be defined as row sections adjacent
to the missing row sections.
[0026] The pixels of the error area may be estimated using an
interpolation technique.
[0027] The image may form an image of a video sequence of images
transmitted in uncompressed or compressed form over a packet
network.
[0028] The image may be a first image of a sequence of images and
the estimated pixels may be used for displaying at least one second
image of the sequence of images, the second image following or
preceding the first image. Accordingly, an inter-image error
estimation may be applied using estimated missing pixels of a
single image, for example in combination with known techniques for
inter-image error concealment.
[0029] A program may have instructions adapted to carry out the
above operations. Further, a computer-readable medium may have a
program embodied therein, wherein the program is to make a computer
execute the above method operations. A computer program product may
comprise the computer-readable medium.
[0030] Further, according to another example, an apparatus for
error concealment in an image includes: a detecting unit for
detecting an error area in image data of an image consisting of
pixels; a determining unit for determining a first boundary section
and a second boundary section of the error area, the boundary
sections including boundary elements being defined based on pixels
with image information close or adjacent to the error area; an
aligning unit for aligning the boundary elements of the first
boundary section and the second boundary section using non-linear
alignment operation to establish correspondences between the
respective boundary elements of the first boundary section and the
second boundary section; and an estimating unit for estimating
pixels of the error area based on the established correspondences
between respective boundary elements of the first boundary section
and the second boundary section.
[0031] Further advantageous features of the invention are described
in further dependent claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0032] FIG. 1 illustrates elements of an apparatus for error
concealment in an image according to an embodiment of the
invention.
[0033] FIG. 2 illustrates operations of a method according to
another embodiment of the invention for error concealment in an
image;
[0034] FIG. 3 shows examples of boundary sections according to
another embodiment of the invention;
[0035] FIG. 4 shows operations of a method for error concealment in
an image according to another embodiment of the invention,
particularly outlining steps for defining ranges of parameters for
boundary sections;
[0036] FIG. 5 shows an example of alignments between pixels of
boundary sections according to another embodiment of the invention;
and
[0037] FIG. 6 illustrates operations for error concealment in an
image according to another embodiment of the invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0038] In the following a first embodiment of the invention will be
described with respect to FIG. 1.
[0039] FIG. 1 illustrates elements of a system for error
concealment in an image, e.g. in a system for compressed video or
image transmission over a communication link.
[0040] FIG. 1 illustrates an exemplary image 110 showing three
error areas 111, 112 and 113. The error areas may include missing
or corrupted pixels. The image may contain grayscale information,
color information or any other type of image information. Further,
the image may constitute a single image or one image of a sequence
of images, e.g., of a video signal. The error areas and the image
itself may be of rectangular shape, as shown in FIG. 1, however,
this is an example only, both the error areas and the image may
have any other shape.
[0041] Further, FIG. 1 shows an apparatus 100 for error concealment
in the image 110, including a detecting unit 101 for detecting an
error area in image data representing an image consisting of a
plurality of pixels. Further, the apparatus 100 includes a
determining unit 102 for determining a first boundary section and a
second boundary section of an error area, the boundary sections
including boundary elements being defined based on pixels with
image information close or adjacent to the error area. For example,
the boundary elements may correspond to boundary pixels with
correct or uncorrupted image information close or adjacent to the
error area, i.e. in the vicinity or in close distance to the error
area. It is, however, also possible that each of the boundary
elements is determined based on a plurality of pixels, for example
in a averaging or weighted averaging operation.
[0042] An aligning unit 103 is provided for aligning the boundary
elements of the first boundary section and the second boundary
section using alignment operations to establish correspondences
between respective boundary elements of the first boundary section
and the second boundary section. For example, linear or non-linear
alignment operations may be used. Finally, the apparatus 100
includes an estimating unit 104 for estimating pixels of the error
area based on the established correspondences between respective
boundary elements of the first boundary section and the second
boundary section.
[0043] After estimating the missing or corrupted pixels of an error
area, the error area may be concealed using the estimated pixels in
order to provide an improved representation of the image.
[0044] The determining unit may be arranged to detect the error
areas 111, 112 and 113 directly in the image, i.e. in an array of
pixels constituting the image, or may detect the error areas in
association with a transmission signal of compressed or
uncompressed image information relating to the image 110, for
example over a packet switched transmission network or any other
communication link.
[0045] Further, FIG. 1 shows an enlarged version of the error area
113. The error area 113 lies adjacent to an image area constituting
a first boundary section 150 including, in the present example,
seven pixels of the image 110 with correct or uncorrupted image
information. The seven pixels of the first boundary section are
denoted 151-157. Each pixel of the first boundary section 150
preferably represents error free image information of the image 110
to be used for estimating the missing or corrupted pixels of the
error area 113. Similarly, the error area 113 lies adjacent to an
image area constituting a second boundary section 160, including
seven boundary pixels 161-167, also preferably including error free
image information of the image 110 to be used for estimating pixels
of the error area.
[0046] Here, the boundary elements are constituted by image pixels,
however, this is an example only, in further examples a boundary
element could be derived from a plurality of pixels of the image,
or of a plurality of pixels of a temporal sequence of images.
[0047] Each of the boundary pixels 151-157 and 161-167 of the first
and second boundary section is represented by at least one
parameter value such as a gray level or color value. As an example,
in FIG. 1 each of the boundary pixels is shown to have a parameter
value indicated by an integer number, which may, for example,
correspond to gray level or color values, as outlined before.
[0048] The aligning unit 103 establishes correspondences between
the boundary pixels of the first boundary section and the second
boundary section using non-linear alignment operations as
illustrated by arrows 171-176 in FIG. 1. The arrows 171-176 show
exemplary established correspondences between the boundary pixels
of the first and second boundary sections based on the parameter
values of the respective pixels indicated by the integer numbers in
the boundary pixels shown in FIG. 1. It is noted that linear
alignment operations may be used as well, and also a combination of
linear and non-linear alignment operations.
[0049] As illustrated, correspondences may be established between
pixels having the same parameter value. Due to the characteristic
of the non-linear aligning operations correspondences may involve
one boundary pixel of each of the first and second boundary section
as illustrated by arrows 171, 172, 175 and 176. However, it is also
possible that a pixel of the first or second boundary section is
defined by the non-linear aligning operations as not corresponding
to any of the pixels of the respective other boundary sections, as
illustrated by pixels 153 and 157 of the first boundary section and
pixel 165 of the second boundary section. Further, a pixel of one
boundary section may be defined to correspond to a plurality of
pixels of the second boundary section as illustrated in FIG. 1 by
arrows 173 and 174 defining a correspondence between pixel 154 of
the first boundary section and the two pixels 163 and 164 of the
second boundary section.
[0050] The non-linear alignment operations for establishing the
correspondences between the pixels of the first and second boundary
sections may include dynamic programming techniques for improved
handling using standard or dedicated digital data processing
devices.
[0051] The system for error concealment shown in FIG. 1 allows to
estimate pixels of error areas in an image or in a sequence of
images in order to conceal the error area when displaying the
image. The concealed error area improves the representation of the
image for a viewer. The non-linear alignment operations allow to
align structural features of the respective first and second
boundary section such that missing features in the error area may
be closely approximated.
[0052] In the following examples of the individual elements of the
system for concealing errors in an image shown in FIG. 1 will be
outlined in further detail.
[0053] In the present embodiment according to the invention, the
detecting unit 101 for detecting error areas, the determining unit
102, the aligning unit 103 and the estimating unit 104 may be
realized involving a computing device such as a general purpose
data processing device or a dedicated computing device for error
concealment. Further, it is also possible that at least one of the
detecting unit, the determining unit, aligning unit and estimating
unit is constituted by a separate data processing device or
hardware device connected to the remaining elements of the
apparatus 100 via a communication link. Moreover, the detecting
unit 101, the determining unit 102, the aligning unit 103 and the
estimating unit 104 may at least partially be constituted by
program sections including instructions for carrying out the
respective functionality of the elements. The program sections may
be executed by a single data processing device, e.g. of a general
purpose computer, or by a plurality of data processing devices
connected to one another for exchange of information. The elements
of the apparatus 100 may partially be realized in hardware and
partially realized in software.
[0054] The detecting unit 101 may be adapted to retrieve image
information from a storage unit or to receive a data stream, e.g.,
including packets of uncompressed or compressed image information,
e.g. over a communication network or any other kind of
communication link.
[0055] Further, the detecting unit may be adapted to detect missing
portions of image information, e.g. missing or corrupted packets
containing image information in order to detect error areas.
Corrupted packets may for example be identified via a CRC checksum
method, as well known in the art, or by any other error detection
method. Missing data packets can be identified using sequential
numbers for consecutive packets, as also well known in the art. Any
other method to identify missing parts of a data stream
representing an image may be employed.
[0056] Moreover, it is possible that the detecting unit 101 detects
error areas in an image by processing or filtering image
information, e.g., to detect missing or corrupted pixels in a
representation of the image, for example as stored as an array in a
memory. The detecting unit may operate on uncompressed image
information or may operate on compressed image information, for
example being compressed using the MPEG-4 video compression
technique or any other compression technique.
[0057] After detecting error areas in an image, the detecting unit
transmits information on the error areas to the determining unit
102 for determining a first boundary section and a second boundary
section of an error area, the boundary sections being defined based
on boundary pixels with correct image information adjacent to the
error area.
[0058] For example, if an error area is constituted by missing
parts of rows of an image, the first and second boundary sections
may include row sections adjacent to the missing row sections, as
for example the first boundary section 150 and the second boundary
section 160 shown in FIG. 1.
[0059] However, it is also possible that the boundary sections are
constituted by column portions, for example to the left or right of
the error area 113 shown in FIG. 1 or may include column sections
and row sections surrounding an error area or any other collections
of pixels, as it will be outlined further below. A boundary section
may include a continuous array of pixels or a discontinuous array
of pixels, e.g. to reduce computing requirements. The first and
second boundary sections may each include an array of pixels
adjacent to the error area located at opposing sides of the error
area or may include arrays of pixels surrounding the error
area.
[0060] An error area may generally have any shape and may include
missing and/or corrupted pixels which need to be concealed. The
determining unit may select uncorrupted image information directly
adjacent to the error area or may select boundary sections in some
distance to the corrupted image information, in order to avoid to
include any corrupted pixels into the boundary sections.
[0061] The first and second boundary section may include different
numbers of pixels, for example the first boundary section could
include a larger number of pixels to be aligned with a smaller
number of pixels of the second boundary section. Since non-linear
alignment operations are used, correspondences between different
numbers of pixels of the boundary sections can be established.
[0062] Further, while the boundary elements of the boundary
sections may be constituted by image pixels, it is also possible
that the boundary sections are constituted by a plurality of pixels
and, optionally, further information. The boundary elements can be
derived from any kind of image information. For example, boundary
elements can be obtained by averaging between two or more adjacent
rows of pixel information in spatial direction or time
direction.
[0063] Further, boundary elements may be obtained by interpolating
between pixel values or by using any other type of image
information, including information on image content, such as
objects, patterns, texture, shapes etc., if available. Thus,
boundary sections may be constituted by a number of boundary
elements, each boundary elements being derived from image
information.
[0064] After the boundary sections are determined, information on
the boundary sections is transmitted to the aligning unit 103 for
aligning the respective pixels of the boundary sections. The
aligning unit uses non-linear alignment operations to establish the
correspondences between the pixels of the first boundary section
and the second boundary section. The alignment operations may
include Needleman-Wunsch techniques for establishing the
correspondences.
[0065] After establishing the correspondences between the pixels of
the boundary sections, as shown in the exemplary embodiment of FIG.
1 using arrows 171-176, information on the established
correspondences is transmitted to the estimating unit 104 arranged
to estimate pixels of the error area between corresponding pixels
of the first boundary section and the second boundary section. Any
method for estimating pixels between two given pixels, i.e.,
boundary pixels may be used, e.g., parameter values of the boundary
pixels may be also assigned to pixels on a connection line between
the boundary pixels, or similar. Examples are a weighted
interpolation or a next neighbor interpolation, allowing an
estimation at reduced computational complexity.
[0066] An interpolation technique, such as a next neighbor
interpolation, constitutes a simple and computationally inexpensive
scheme for estimating pixels of the error area between
corresponding pixels of the boundary sections.
[0067] As a more elaborate method for estimating pixels of the
error area between corresponding pixels of the boundary sections, a
weighted interpolation between the boundary pixels can be chosen,
leading to improved results. It is noted that any known technique
may be used for estimating the pixels of the error area between
corresponding pixels of the boundary sections.
[0068] The estimating unit may be further adapted to introduce the
estimated pixels into the image, such as the estimated pixels of
the error area 113 into the image 110 for providing a
representation of the image 110 with concealed errors.
[0069] As outlined above, the operations for error concealment in
an image may be realized at least partially in software, e.g. by
programs or program modules.
[0070] Further, a computer-readable medium may be provided having a
program embodied thereon, where the program is to make a computer
or a system of data processing devices execute functions or
operations of the features and elements of the above-described
embodiment of the invention.
[0071] A computer-readable medium can be a magnetic or optical or
other tangible medium on which a program is recorded, but can also
be a signal, e.g., analog or digital, electromagnetic or optical,
in which the program is embodied for transmission.
[0072] Further, a computer program product may be provided
comprising the computer-readable medium.
[0073] The digital data of the image or sequence of images may be
temporarily stored in a memory section (not shown) connected to the
apparatus 100 or may be stored externally in a database. However,
it is also possible, for example in an internet application, that
the image data are transmitted over a network to a client computing
device for displaying the image or sequence of images after
performing the error concealment according to the invention. The
image or video information may be received in compressed or
uncompressed form.
[0074] In the following a further embodiment of the invention will
be described with respect to FIG. 2.
[0075] FIG. 2 shows operations of a method for error concealment in
an image according to another embodiment of the invention. The
operations shown in FIG. 2 may be carried out using the apparatus
shown in FIG. 1, however, FIG. 2 is not limited thereto.
[0076] In a first operation S201 an error area is detected in an
image, the image being constituted by a plurality of pixels. As
outlined before, the image may also be part of a sequence of
images, for example of a video sequence. The image may be received
in uncompressed or compressed form over a communication link.
[0077] Error areas may be detected using a detection unit such as
the detection unit 101 outlined in FIG. 1. An error area may
include corrupted or missing pixels of the image. As an image may
include a plurality of error areas, it is possible that at least
two error areas are combined to form a composite error area, e.g.,
if only few pixels are available in a larger area, the entire area
may be handled as an error area.
[0078] Any error detection algorithm may be applied to detect
corrupted pixels of the image and to detect missing pixels in the
image or a data stream constituting the image.
[0079] After detecting the error area in the image, in an operation
S202 a first boundary section and a second boundary section
adjacent to the error area are determined, the boundary sections
being defined based on boundary pixels with correct image
information.
[0080] As outlined before, while the boundary elements may be
constituted by image pixels, it is also possible that the boundary
sections constitute values being defined based on image pixels.
Further, the first and second boundary sections may lie directly
adjacent to the error area or in some distance thereto, e.g., in
order to avoid including corrupted pixels into the boundary
sections. Further, a boundary section may be constituted by a
continuous or discontinuous array of pixels, e.g. a row section or
any other shape, including curved or angular shapes.
[0081] The boundary sections may be defined at opposing edges of
the error area, or may be defined to surround the error area, in
which case an end portion of a boundary section may abut a starting
portion of the second boundary section.
[0082] Thereafter, in an operation S203 the boundary pixels of the
first and second boundary sections are aligned using non-linear
alignment operations, in order to establish correspondences between
respective pixels of the boundary sections. Dynamic programming
techniques may be used for convenient execution of the error
concealment algorithms on a data processing device. As outlined
before, the correspondences may include a 1 to 1 correspondence,
may define a pixel of a boundary section to not correspond to any
pixel of the other boundary section, or may define multiple
correspondences between one boundary pixel of the first boundary
section and a plurality of boundary pixels of the second boundary
section.
[0083] Thereafter, in an operation S204 the pixels of the error
area are estimated based on the established correspondences, for
example using a next neighbor interpolation or a weighted
interpolation or any other interpolation technique. The estimated
pixels are then used for generating an error concealed
representation of the image.
[0084] It is also possible, if the image constitutes a first image
of a sequence of images, that the estimated pixels are used for
displaying at least one second image of the sequence following or
preceding the first image. This can be particularly useful in video
applications using compression techniques, where errors in one
frame may also affect adjacent frames, i.e., errors in one frame
may propagate through a plurality of frames.
[0085] Further, if the image constitutes an image of a sequence of
images, such as of a video sequence, an error area may not only be
present in a single image, but may be repetitively present in a
sequence of images.
[0086] In this case it is also conceivable to define boundary
sections not only in the two spatial dimensions such as outlined
above in a single image, but also to define boundary sections in
three dimensions, i.e. the third dimension considering the temporal
sequence of images.
[0087] For example, if an error region is present in one image or
at a substantially invariant location in a plurality of images of a
sequence of images, the first boundary section may be defined in
the image preceding the error affected image or images, e.g., the
last image in a sequence before the error area appears. Further,
the second boundary section could be defined in an image subsequent
to the last image involving the error area. Thus, the first
boundary section and the second boundary section would be defined
in two images of a video sequence framing a subsequence of images
affected by errors.
[0088] Further, in the above example, it is also possible that a
boundary section is defined as a three dimensional array of
pixels.
[0089] A boundary section may also extend in temporal direction in
a plurality of images of the sequence of images.
[0090] Still further, it is noted that boundary sections may be
derived from a plurality of images of a temporal sequence of
images, as in this case noise components can be eliminated. A
boundary section can be determined by averaging pixels of a
sequence of images, performing a weighting operation, e.g. applying
a high weighting factor to pixels of an image featuring the error
area, whereas images temporally before or after the image with the
error area may be associated with lower weight factors.
[0091] Similarly, boundary sections could be defined in a
multidimensional alignment using several color channels of a
luminance channel and several chrominance or color channels.
[0092] Boundary sections could be also defined by combining the
above approaches, i.e., in an approach using a time sequence of
information from different channels such as color, luminance
channels, etc.
[0093] In the following an example for aligning the boundary pixels
of a first boundary section and a second boundary section using
non-linear alignment operations to establish correspondences
between respective pixels of the first boundary section and the
second boundary section will be outlined with respect to a further
embodiment.
[0094] In this example, it is assumed that dynamic programming
techniques using Needleman-Wunsch techniques are applied.
[0095] For the sake of convenience in the present example
comparatively short boundary sections are used as examples.
Further, in the present case only four different parameter values
describing the boundary pixels are assumed to be present. These
parameter values are represented by the integer values 1, 2, 3 and
4. For example the parameter values may correspond to gray level
information, color information or similar, as outlined before.
[0096] It is, however, also possible that the parameter values
correspond to a range of gray levels or color values, in order to
allow a substantially reduced number of possible parameter values
for reduced computational complexity, as outlined further
below.
[0097] Of course, in practical examples, a larger number of
parameter values may be present, and further, a larger number of
pixels of the respective boundary sections may be present.
[0098] In this example it is assumed that a first boundary section
determined for example by the determining unit 102 outlined with
respect to FIG. 1 includes eleven pixels with the following
individual parameter values:
[0099] 21144312441 (first boundary section)
[0100] Further, it is assumed that the second boundary section
includes seven pixels with the following distribution of parameter
values:
[0101] 2214321 (second boundary section)
[0102] Accordingly, the length M of the first boundary section is
M=11 and the length N of the second boundary section is N=7.
[0103] For example, in the present embodiment the first and second
boundary sections may be constituted by row sections or by any
other continuous or discontinuous array of pixels.
[0104] For the dynamic programming approach applied in the present
embodiment, a simple scoring scheme is assumed wherein
[0105] S.sub.ij=1, if the parameter value at position i of the
first boundary section (that is the i-th pixel of the first
boundary section) is the same as the parameter value of position j
of the second boundary section (i.e., the j-th pixel of the second
boundary section). Si,j constitutes a match score. Otherwise
[0106] S.sub.i,j=0, which constitutes a mismatch score.
[0107] w=0, constitutes a gap penalty applied for generating a gap
in the established sequences of correspondences between the pixels
of the first boundary section and the second boundary section.
[0108] It is noted that it is optional to replace the equality of
the matrix fill step by a similarity by setting S.sub.ij=1,
equality of the matrix fill step may also be assumed for pixel
values not too far apart.
[0109] The dynamic programming approach of the present embodiment
includes an initialization step, a matrix fill operation, i.e., a
scoring operation and a trace back operation, i.e. an alignment
operation.
[0110] First, the operations of the initialization step will be
outlined in detail.
[0111] The first step in the global alignment dynamic programming
approach is to create a matrix with M plus one columns and N plus
one rows, where M and N correspond to the sides of the boundary
sections to be aligned, as outlined above.
[0112] For simplicity reasons in the present example it is assumed
that there is no gap opening or gap extension penalty to be applied
and therefore the first row and first column of the matrix can be
filled with zero.
1TABLE 1 1
[0113] In the following step the operations of the matrix fill step
are outlined in further detail.
[0114] A possible solution for the matrix fill operation finds the
maximum global alignment score by starting in the upper left-hand
corner in the matrix shown in Table 1 and finding the maximal score
M.sub.i,j for each position in the matrix. In order to find
M.sub.i,j for any i,j, it is minimal to know the score for the
matrix positions to the left, above and diagonal to i,j. In terms
of matrix positions, it is therefore necessary to know M.sub.i-1,j,
M.sub.i,j-1 and M.sub.i-1,j-1.
[0115] For each position, M.sub.i,j is defined to be the maximum
score at position i,j; i.e.
[0116] M.sub.i,j=MAXIMUM[
[0117] M.sub.i,j, j-1+S.sub.i,j (match/mismatch in the
diagonal),
[0118] M.sub.i,j-1+w (gap in first boundary section),
[0119] M.sub.i-1,j+w (gap in second boundary section)]
[0120] Using this information, the score at position 1,1 in the
matrix can be calculated since the first parameter value in both
sequences is a 2, S.sub.1,1=1, and by the assumptions stated at the
beginning, w=0. Thus,
M.sub.1,1=MAX[M.sub.0,0+1,M.sub.1,0+0,M.sub.0,1+0]=MAX[1,0,0]=1.
[0121] A value of 1 is then placed in position 1,1 of the scoring
matrix, as outlined in Table 2.
2TABLE 2 2
[0122] Since in the present example it is assumed that the gap
penalty w=0, the rest of row 1 and column 1 can be filled in with
the value 1.
[0123] It is, however, noted that the gap penalty w=0 is an example
only, chosen in the present example. Any gap penalty unequal zero
can be defined.
[0124] For example row 1 is considered, at column 2 the value is
the max of 0 (for a mismatch), 0 (for a vertical gap) or 1 (for a
horizontal gap). The rest of row 1 can then be filled out similarly
until column 8 is reached. At this point there is a parameter value
2 encountered in both sequences. Thus, the value for the cell at
row 1, column 8 is the maximum of 1 (for a match), 0 (for a
vertical gap) or 1 (for a horizontal gap). The value will again be
1. The rest of row 1 and column 1 is filled with 1 using the above
reasoning, as shown in Table 3.
3TABLE 3 3
[0125] Now attention is drawn to column 2. The location at row 2
will be assigned the value of the maximum of 1 (a mismatch), 1 (a
horizontal gap) or 1 (a vertical gap). So the value to be filled in
is 1.
[0126] At the position of column 2, row 3, there is a parameter
value 1 in both sequences. Thus, its value will be the maximum of 2
(match), 1 (horizontal gap), 1 (vertical gap) and thus the value to
be entered is 2.
[0127] Moving along to position column 2, row 4, the value to be
entered at this location will be the maximum of 1 (mismatch), 1
(horizontal gap), 2 (vertical gap) and thus, the value to be
entered is 2. Note that for all of the remaining positions except
the last one in column 2, the choices for the value will be the
exact same as in row 4, since there are no matches. The final row
will contain the value 2, since it is the maximum of 2 (match), 1
(horizontal gap) and 2 (vertical gap).
[0128] The filled in column is shown in Table 4.
4TABLE 4 4
[0129] Using the same techniques as described for column 2, the
matrix locations in column 3 are filled. The result, using the
above technique, is shown in Table 5.
5TABLE 5 5
[0130] After filling in all of the values, the score matrix reads
as shown in Table 6.
6TABLE 6 6
[0131] Table 6 shows the result of the matrix fill in step and thus
the operations proceed to the traceback step.
[0132] In the traceback step, the maximum alignment score for the
two exemplary boundary sequences is 6. The traceback step then
determines the actual alignments that result in the maximum score.
It is noted that with a simple scoring algorithm such as the one
that is outlined here, there are likely to be multiple maximal
alignments.
[0133] The traceback step begins at the M,N position in the matrix,
i.e., the position that leads to the maximal score. In this case
there is a 6 in that location.
[0134] The traceback step takes the current cell and looks to the
neighbor cells that could be direct predecessors. This means, the
traceback operations require to look to the neighbor to the left
(gap in the second boundary section), the diagonal neighbor
(match/mismatch) and the neighbor above (gap in the first boundary
section). The algorithm for the traceback requires to choose as the
next cell in the sequence one of the possible predecessors. In this
case, the neighbors are all equal 5 as shown in Table 7.
7TABLE 7 7
[0135] Since the current cell has a value of 6 and the score is a 1
for a match and 0 for anything else, the only possible predecessor
is the diagonal match/mismatch neighbor. If more than one possible
predecessor exist, any can be chosen. The above operations yield a
current alignment of
[0136] (first boundary section) 1 (parameter value)
[0137] (second boundary section) 1 (parameter value)
[0138] After establishing this alignment, attention is drawn to the
next current cell and it is determined which cell is to be chosen
as its direct predecessor. In this case, as the maximum scores are
4 and 5, the cell to the left of the current cell with the value 5
is to be chosen, as shown in Table 8.
8TABLE 8 8
[0139] Thus, the above alignment results a gap to the second
boundary section, so the current alignment is 9
[0140] In the next step, once again the direct predecessor produces
a gap in the second boundary section as shown in Table 9.
9TABLE 9 10
[0141] After this step the current alignment becomes 11
[0142] Continuing with the traceback operations as outlined above,
the operations eventually reach a position in column 0, row 0,
which indicates that the traceback step is completed. One possible
maximum alignment can be defined as indicated in Table 10.
10TABLE 10 12
[0143] which results in the following alignment of the pixels of
the first and second boundary sections: 13
[0144] Of course, this alignment of the boundary pixels is only one
possible solution, an alternate solution may be obtained as shown
in Table 11.
11TABLE 11 14
[0145] resulting in an alignment of the boundary pixels as follows:
15
[0146] There are further alternative solutions each resulting in a
maximum global alignment score of 6. Since this behaves
exponentially, the dynamic programming operations may involve
selecting only a single solution.
[0147] The above example shows a simple aligning scheme, and it is
noted that advanced such aligning schemes may be used to perform
the alignment operation of the pixels of the first boundary section
and the second boundary section, as known in the art.
[0148] Parameters used in the alignment procedure can be determined
at the sender and signaled, e.g. for each packet of data the
parameters can be signaled in a following packet.
[0149] In the following a further embodiment of the invention will
be described with respect to FIG. 3.
[0150] FIG. 3 shows various examples of error areas identified in
an image such as image 110 described with respect to FIG. 1.
[0151] As noted before, the boundary sections identified can have
any given shape and can be constituted by a continuous or
discontinuous array of pixels.
[0152] With respect to FIG. 1 examples of error areas having
rectangular shape were described, and, boundary section lying
adjacent to the upper portion of the error area in image 110 and
the lower portion of the error area were defined. As the image is
constituted by rows of image information, the first and second
boundary sections therefore were constituted by sections of rows
containing uncorrupted image information.
[0153] However, as shown on the left-hand side of FIG. 3, it is
also possible to define column portions as first and second
boundary sections 301 and 302, lying adjacent to an error area 303.
In this case, the individual pixels of the boundary sections 301
and 302 are established and pixels of the error area can be
estimated. Even though in the present case the number of pixels of
the first and second boundary sections are the same, as also in the
embodiment described with respect to FIG. 1, it is noted that the
first and second boundary section may include different number of
pixels, e.g., the second boundary section may include a smaller
number of pixels than the first boundary section.
[0154] Further, in FIG. 3 a second example of an error area is
shown in the middle of the figure, denoted 310. This error area is
a rectangular error area with a protruding portion in the middle
section.
[0155] In this example, a first boundary section 311 is considered
to be defined by the pixels lying adjacent to the error area 310 in
the "upward" direction, forming a discontinuous array of pixels.
The second boundary section is constituted by a row portion of
boundary pixels 312, similar to previous embodiments.
[0156] In this case, correspondences will be established, as
before, between the individual pixels of the first boundary section
and the second boundary section for estimating pixels of the error
area 310 with the protruding portion.
[0157] It is noted that apart from the shown examples, the error
areas may have further shapes including triangular shapes, circular
shapes, and any other shapes.
[0158] On the right-hand side of FIG. 3 a third example of an error
area and boundary sections is illustrated. In this case, an error
area 320 is considered to be constituted by a rectangular region,
for example as region 113 shown with respect to FIG. 1.
[0159] In the example shown on the right side of FIG. 3, it is
assumed that a first boundary section 321 is defined as the pixels
surrounding the error area 320 in the "upward" direction and to the
"right"-hand side of the error area.
[0160] Further, a second boundary section 322 is considered to be
constituted by the pixels surrounding the error area 320 in
"downward" direction and to the "left-hand side". Thus, in the
present example, the first and second boundary sections 321 and 322
fully enclose the error area and lie adjacent to one another at the
left lower corner of the error area and the upper right corner of
the error area. Accordingly, the first boundary section is
constituted by a row portion and by a column portion of pixels and
similarly the second boundary section is constituted by a row
portion and a column portion of pixels of the image.
[0161] As before, correspondences are established between the
pixels of the first boundary section and the second boundary
section, which may involve establishing correspondences between
pixels of the respective row portions of the first and second
boundary sections, as indicated by a double-arrow 324, and may
include establishing correspondences between pixels of a row
section of the first boundary section and a column section of the
second boundary section, as illustrated by a double-arrow 323.
[0162] In further examples, the first and second boundary sections
such as 321 and 322 may include different numbers of pixels, as
outlined before.
[0163] Further, it is also possible that boundary sections overlap
each other.
[0164] The examples outlined with respect to FIG. 3 illustrate some
possible techniques for determining boundary sections, e.g.,
allowing to adapt the selection of boundary sections to image
characteristics such as texture, known graphic elements and
similar.
[0165] In the following a further embodiment of the invention will
be described with respect to FIG. 4.
[0166] FIG. 4 outlines operations for defining and grouping image
parameters as for example gray level values, color level values and
similar in preparation of performing the alignment operations for
example as outlined with respect to the previous embodiments. The
operations of FIG. 4 may be carried out using the system shown in
FIG. 1, however, FIG. 4 is not limited thereto.
[0167] In a first operation S401 image parameters such as gray
level, color level and similar are defined, preferably in
dependence on the encountered pixel values of the image under
consideration. Thus, an image parameter may represent one of gray
level, color level and similar or a combination of these
values.
[0168] In today's imaging applications parameter values of pixels
of an image may have a wide range, for example, they may be
constituted by 8 bit values, 16 bit values, or 32 bit values or
even values having higher bit numbers. Performing non-linear
alignment operations according to the invention for pixels having a
wide range of parameter values can become computationally
complex.
[0169] Therefore, in the non-linear alignment operation, in order
to reduce the computational complexity, the number of different
values for the parameter values can advantageously be reduced by
introducing a similarity measure for grouping pixels having certain
"similar" parameter values. For example, the similarity measure may
include classifying the boundary pixels into a plurality of ranges
for the pixel parameters, e.g. before a matrix fill operation such
as outlined with respect to FIG. 2. And, the width of the
individual ranges may correspond to a distribution of the parameter
values.
[0170] As a practical example, the allowable computational
complexity may allow for a number of X different parameter values
for the individual pixels of the image, and therefore the available
range of the parameter values may be subdivided into X sub-ranges,
each of the X ranges including a sub-portion of the image parameter
values. Thus, the similarity measure may be used for grouping
pixels having similar gray levels or color levels or similar.
[0171] With the number of image parameter values the computational
complexity can be adjusted and adapted to the available
resources.
[0172] As outlined above, the similarity measure may include
classifying the boundary pixels into a plurality of ranges of pixel
gray level values, color values or similar.
[0173] Accordingly, the exemplary parameter values 1, 2, 3, 4 used
in the example described with respect to FIG. 2 may each constitute
a sub-range of gray levels, color values or similar.
[0174] Still further, the individual sub-ranges may be adapted to
an encountered image specific distribution of the image gray
levels, color values or similar, and therefore, in operation S401
it may further be determined a distribution of image parameter
values.
[0175] The above operation S401 will be further outlined by way of
an example.
[0176] It is assumed that an image subjected the inventive error
concealment is constituted by a gray level representation of an
object. The object is considered to have generally dark features,
and, an error area is assumed to be present in one of the dark
features of the object. Therefore, the boundary pixels surrounding
the error area will also have generally dark gray levels and the
distribution of image parameter values for the non-linear alignment
operation should reflect the generally dark appearance of the
boundary pixels.
[0177] Thus, in an operation S402 the width of the parameter ranges
can advantageously be determined in correspondence to the
distribution of encountered pixel values, i.e. the encountered
pixel values of the boundary sections.
[0178] In an operation S403 the boundary pixels are then classified
into the plurality of ranges of pixel parameter values defined in
operation S402. Accordingly, a first range may include a large
number of parameter values, whereas a second parameter sub-range
may include only a small number of possible parameter values.
[0179] Thereafter, in an operation S404 the dynamic programming
operations, Needleman-Wunsch techniques and similar, constituting
the non-linear alignment operations are applied using the defined
parameter ranges, as outlined before. In other words, the
individual parameter values described with respect to previous
embodiments may correspond to a parameter value sub-range.
[0180] Defining the individual width of individual ranges in
correspondence to a distribution of pixel values such as gray level
and color level allows to adapt the classification operation to
particular image characteristics.
[0181] In the following a further embodiment of the invention will
be described with respect to FIG. 5.
[0182] FIG. 5 shows a further example of correspondences between
pixels of the first and second boundary section.
[0183] FIG. 5 shows an error area 500, as for example corresponding
to the error area 113 shown with respect to the image 110
illustrated in FIG. 1. It is assumed that a first boundary section
510 including boundary pixels 511, 512-519 are defined. Further, it
is assumed that a second boundary section 520 including boundary
pixels 521-529 is defined.
[0184] In the present example it is assumed that correspondences
are established between the individual pixels of the first and
second boundary sections as illustrated by arrows 531, 532-540.
[0185] While in the example outlined with respect to FIG. 1
correspondences were only established between pixels having
identical parameter values, as shown by arrows 531, 532, 533, 534,
536, 537, 539 and 540. However, in the present example, the
non-linear alignment operation is modified such that
correspondences can also be established between pixels having
parameter values similar to one another, similar to what was
outlined with respect to FIG. 4. Alignments between pixels having
different parameter values are illustrated by arrows 535 and
538.
[0186] Allowing an alignment of pixels having different parameter
values can be regarded as an alternative to the embodiment outlined
with respect to FIG. 4, where the parameter values were grouped
into sub-ranges.
[0187] In the present example the parameter values can correspond
to the "full resolution" image information, and the above
similarity measure may be applied in the non-linear alignment
operation, allowing alignments between pixels having "similar"
parameter values. As before, the similarity measure may include
classifying the boundary pixels into a plurality of ranges of pixel
parameter values, and, the width of the individual ranges with
allowed correspondences may be established in connection with a
distribution of the parameter values.
[0188] In the following a further embodiment of the invention will
be described with respect to FIG. 6.
[0189] FIG. 6 illustrates operations of a method for error
concealment in an image. The operations shown in FIG. 6 may be
carried out using the apparatus shown in FIG. 1, however, FIG. 6 is
not limited thereto.
[0190] The embodiment shown in FIG. 6 may be applied to an
application which receives decompressed image information, for
example over a network such as the Internet. For example, a single
image or a video sequence of images in a compressed format such as
the MPEG-4 format or any other compressing format can be
considered.
[0191] In a first operation S601 the image information is received
and appropriately decompressed, using a decompression algorithm, as
known in the art.
[0192] In an operation S602 error areas are identified in the
image, if present, as for example outlined with respect to previous
embodiments. The error area detection may make use of existing
algorithms such as CRC check codes or operations for identifying
packet loss in packet switched transmission networks, such as the
TCP/IP protocol.
[0193] In an operation S603 it is determined whether there are any
error areas not yet concealed. If in operation S603 the decision is
NO, i.e., unconcealed error areas are still present, in an
operation S604 in the image data the error areas are detected. The
error areas may include a plurality of error pixels containing
incorrect image information or may be constituted by missing pixels
of the image, for example if entire packets with image information
were lost during transmission.
[0194] In an operation S605 a first and second boundary section
adjacent to the error area is detected, each including an array of
pixels adjacent to the error area and located at opposing sides of
the error area, as for example outlined with respect to previous
embodiments.
[0195] In an operation S606 correspondences from each pixel of the
first boundary section to no pixel or at least one pixel of the
second boundary section are established as particularly outlined
with respect to FIGS. 1, 3 and 5.
[0196] Thereafter, in an operation S607 the boundary pixels of the
boundary sections are aligned using non-linear alignment
operations, to establish correspondences between the respective
pixels of the boundary sections, as also outlined with respect to
previous embodiments.
[0197] In an operation S608 the pixels of the error area are
estimated based on a weighted interpolation, a next neighbor
interpolation or any other interpolation or estimation technique,
as outlined before or known in the art.
[0198] Thereafter, in an operation S609, the pixels of the error
area are filled with the estimated pixels, in preparation of
displaying the image. Thereafter the flow returns to step S602, and
it is searched for a further error area.
[0199] If in operation S603 it is determined that no further error
area can be determined, i.e., the image error is now error free,
the image can be displayed in a step S601, or transmitted to
further elements of the system for further processing.
[0200] Thereafter the flow ends.
[0201] According to another embodiment an apparatus for error
concealment in an image may have the following constitution.
[0202] 1). Apparatus for error concealment in an image,
including:
[0203] a code section containing instructions to detect an error
area in image data of an image consisting of pixels;
[0204] a code section containing instructions to determine a first
boundary section and a second boundary section of the error area,
the boundary sections including boundary elements being defined
based on pixels with image information close to the error area;
[0205] a code section containing instructions to align the boundary
elements of the first boundary section and the second boundary
section using alignment operations to establish correspondences
between respective boundary elements of the first boundary section
and the second boundary section; and
[0206] a code section containing instructions to estimate pixels of
the error area based on the established correspondences between
respective boundary elements of the first boundary section and the
second boundary section.
[0207] 2). Apparatus of 1), including a code section containing
instructions for executing dynamic programming techniques.
[0208] 3). Apparatus of 1), wherein the alignment operations
include Needleman-Wunsch techniques.
[0209] 4). Apparatus of 1), including a code section containing
instructions performing a matrix fill operation using a similarity
measure.
[0210] 5). Apparatus of 1), including a code section containing
instructions to classify the boundary elements into a plurality of
ranges of pixel parameters, a pixel parameter being constituted by
at least one of grey level values and colour values.
[0211] 6). Apparatus of 5), including a code section containing
instructions to define the width of the individual ranges
corresponding to a distribution of the parameter values.
[0212] 7). Apparatus of 1), including a code section containing
instructions to define the boundary elements based on at least one
image pixel.
[0213] 8). Apparatus of 1), including a code section containing
instructions to establish of correspondences from a boundary
elements of the first boundary section to no boundary elements or
at least one boundary elements of the second boundary section.
[0214] 9). Apparatus of 1), including a code section containing
instructions to estimate the pixels of the error area using at
least one of a weighted interpolation and a next neighbour
interpolation.
[0215] 10). Apparatus of 1), wherein the image is a first image of
a sequence of images, and including a code section containing
instructions to use the estimated pixels or displaying at least one
second image of the sequence, the second image following or
preceding the first image.
* * * * *